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Abstract 


We  present  an  analysis  of  an  expert  performing  a  highly  interactive  computer  task.  The 
analysis  uses  GOMS  n  models,  specifying  the  Goals,  Operators,  Methods,  and  Selection 
rules  used  by  the  expert;  the  GOMS  models  are  implemented  within  an  unified  theory  of 
cognition  called  Soar.  Two  models  are  presented,  one  mthfunction-level  operators  which 
perform  high-level  functions  in  the  domain,  and  one  with  keystroke-level  operators  which 
describe  hand  movements.  For  a  segment  of  behavior  in  which  the  expert  accomplished 
about  30  functions  in  about  30  second the  function-level  model  predicted  the  observed 
behavior  well,  while  the  keystroke-level  model  predicted  only  about  half  of  the  observed 
hand  movements.  These  results,  including  the  discrepancy  between  the  models,  are 
discussed. 
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MOTIVATION 

This  research  was  done  in  preparation  for  a  panel  at  the  Human  Factors  Society  (HFS) 
Meeting  in  Orlando,  Oct.  5-12, 1990,  organized  by  Wayne  Gray  and  Mike  Atwood  of  the 
Intelligent  Interfaces  Group  at  the  NYNEX  Science  and  Technology  Center.  The  purpose 
of  the  panel  was  to  demonstrate  GOMS  analyses  to  the  human  factors  (HF)  community  in 
an  exciting  manner  so  that  HF  practitioners  would  be  motivated  to  learn  more  about  GOMS 
and  use  it  in  their  work. 


Three  applied  psychology  researchers  (Judy  Olson  of  the  University  of  Michigan,  Jay 
Elkerton,  of  Philips  Laboratories,  and  Bonnie  John  of  Carnegie  Mellon  University)  agreed 
to  analyze  a  task  selected  by  Wayne  Gray.  Gray  selected  a  human-computer  interaction 
domain,  made  a  videotape  of  an  expert  interacting  in  that  domain,  and  gave  the  videotape 
and  an  associated  transcription  to  each  researcher  (Appendix  I).  In  the  interest  of  enticing 
HFS  conference  participants  to  the  panel,  and  stretching  the  GOMS  methodology  to  include 
real-time,  interactive  domains,  Gray  selected  a  domain  hitherto  unanalyzed  with  GOMS: 
video  game  play  (in  particular,  Nintendo's  Super  Mario  Bros.  31  adventure  game).  The 
four  panelists  (the  three  previously  named  researchers  and  Wayne  Gray,  himself)  each  did 
a  GOMS  analysis  of  the  interaction  independently,  and  presented  the  results  to  each  other, 
and  to  the  HFS  audience,  for  the  first  time  during  the  panel  session.  Because  of  limited 
presentation  time,  the  panelists  restricted  their  analyses  to  the  first  27  seconds  of  the 
expert's  interaction  (a  natural  break  in  the  play,  where  the  expert  flies  the  game's  main 
character,  Mario,  off  the  screen  and  to  a  different  part  of  the  game's  world).  The  panel 
continued  with  a  discussion  of  the  similarities,  differences,  and  uses  of  the  analyses. 
Informal  comments  to  the  panelists  indicate  that  the  attendees  were  indeed  excited  by  the 
presentations  and  were  motivated  to  learn  more  about  GOMS,  with  the  hope  of  using  it  in 
their  work. 


Our  own  research  interests  lie  in  the  creation  of  engineering  models  of  computer  users  that 
allow  quantitative  prediction  of  hard  measures  (performance  time,  learning  time,  the 
commission,  detection ,  and  repair  of  errors,  etc.)  in  the  specification  stage  of  system 
design  (see  Newell  &  Card,  1985, 1986;  John,  1988  for  detailed  discussions  of  this 
position).  We  are  currently  extending  GOMS  to  highly  interactive  task  domains.  By  highly 
interactive  tasks,  we  mean  tasks  in  which  a  user  perceives  a  display,  comprehends  and 
responds  to  that  display  on  the  order  of  once  every  second,  for  several  seconds  at  a  time. 
We  call  this  rapid  interaction  the  immediate  interaction  cycle.  Tasks  displaying  the 
immediate  interaction  cycle  are  common  in  human-computer  interaction  (HQ):  searching 
for  information  with  computer  browsers,  constructing  a  new  diagram  with  an  interactive 
graphics  package,  playing  "what-if '  with  a  spreadsheet,  creating  a  schedule  with  PERT 
charts,  exploring  mathematical  relationships  with  graphical  statistics  programs,  creating 
real-time,  multi-media  presentations.  The  list  seems  endless.  Video  games  are  extreme 
examples  of  such  tasks.  The  interaction  in  many  video  games  seems  almost  manically 
driven  by  the  game  rather  than  by  the  player.  Goals  seem  to  be  interrupted,  suspended  and 
returned  to.  Yet,  similar  to  other  domains  successfully  modelled  by  GOMS,  an  expert's 
behavior  looks  to  be  highly  knowledge  intensive  and  eventually  seems  to  become  routine. 
We  welcomed  the  opportunity  to  stretch  our  analysis  techniques  to  the  challenge  of  Super 
Mario  Bros.  3. 
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1  Super  Mario  Bros.  3,  Nintendo,  and  the  games  and  characters  discussed  in  this  paper  are  trademarks  of,  and 
under  copyright  to,  Nintendo  of  America,  Inc. 
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This  technical  report  provides  documentation  of  the  analyses  presented  by  Bonnie  John  at 
that  HFS  panel.  The  content  of  the  analyses  remains  essentially  the  same.  At  the  time  of 
the  presentation,  many  assumptions  and  decisions  made  were  made  from  intuitions  bom  of 
intimate  knowledge  of  HOMS  analyses.  Here,  we  present  the  logic  underlying  those  initial 
intuitions.  We  hope  this  report  will  further  encourage  the  panel's  attendees,  and  others  in 
the  fields  of  applied  psychology  and  cognitive  modeling,  to  learn  more  about  the  GOMS 
methodology,  and  the  extensions  to  it  made  with  the  Soar  unified  theory  of  cognition 
described  herein. 

fi.QMS_M-QD.ELS 

GOMS  is  a  formalism  for  representing  routine  cognitive  skill.  The  original  concept 
appeared  in  The  Psychology  of  Human-Computer  Interaction ,  by  Card,  Moran,  and 
Newell  (1983,  and  also  in  an  earlier  article,  Card,  Moran,  &  Newell,  1980)  as  a  model  of 
the  performance  of  computer  users.  GOMS  stands  for  Goals,  Operators,  Afethods,  and 
Selection  rules.  A  goal  is  a  symbolic  stt  ucture  that  defines  a  state  to  be  achieved,  and 
determines  a  set  of  methods  by  which  it  .-nay  be  accomplished.  Operators  are  elementary 
perceptual,  cognitive,  or  motor  acts,  whose  execution  is  necessary  to  change  any  aspect  of 
the  user's  mental  state  or  to  affect  the  task  environment.  An  operator  is  defined  by  a 
specific  effect  (output)  and  by  a  specific  duration.  An  operator  may  take  inputs,  and  its 
outputs  and  duration  may  be  a  function  of  its  input: .  The  selection  of  operators  for  any 
specific  GOMS  model  defines  its  grain  of  analysis.2  Methods  are  procedures  for 
accomplishing  a  goal.  A  method  is  a  sequence  of  goals  and  operators,  with  conditional 
tests  on  the  contents  of  the  user's  immediate  memory  and  on  the  state  of  the  task 
environment  In  routine  cognitive  tasks,  methods  are  assured  of  success  (up  to  the 
possibility  of  having  been  mis-selected,  errors  in  implementation,  and  the  reliability  of  the 
equipment).  By  contrast,  in  problem-solving  tasks,  methods  may  or  may  not  lead  to 
success  depending  on  the  user’s  lack  of  knowledge  or  understanding  of  the  task 
environment.  Also,  GOMS  methods  are  procedures  that  the  user  already  has  at 
performance  time,  as  opposed  to  plans  created  during  task  performance.  If  more  than  one 
method  can  be  used  to  accomplish  a  goal,  a  selection  must  be  made.  The  essence  of  skilled 
behavior  is  that  these  selections  are  not  problematic,  that  they  proceed  quickly  and 
smoothly.  Although  the  original  description  of  GOMS  acknowledges  that  extended  (but 
still  unproblematic)  decision  processes  may  be  necessary  in  some  situations,  selection 
rules ,  which  are  if-then  rules  that  recognize  the  method  appropriate  for  the  specific  task 
situation,  are  a  signature  of  routine  cognitive  skill  and  GOMS. 

This  original  specification  of  GOMS  had  many  limitations.  Among  them,  GOMS  predicted 
only  error-free  behavior  with  no  mechanisms  for  predicting  the  occurrence  of  errors, 
neither  frequency,  type,  nor  when  they  might  occur.  There  was  no  mechanism  for 
learning.  The  goal-stack  control  structure  was  inadequate  for  handling  interruptions. 
Operators  in  the  original  GOMS  model  were  on  the  order  of  one  half  second,  at  their 
smallest  grain  of  analysis,  and  many  contained  mixtures  of  perceptual,  cognitive  and  motor 
acts.  This  confounding  of  processes  dictated  that  operators  be  sequential,  whereas  many 
skilled  tasks  exhibit  concurrent  perceptual,  cognitive,  and  motor  acts. 

Although  not  limited  by  the  architecture  of  GOMS,  the  example  tasks  in  the  1983  volume 
implicitly  defined  boundaries  of  GOMS  analyses.  These  early  tasks  had  a  static  visual 


2  Card  et  al.  (1983,  Chaplet  5)  demonstrate  GOMS  models  of  text-editing  at  four  levels  of  analysis,  from 
the  unit  task  level  where  each  editing  modification  (e.g.,  delete  a  paragraph)  is  accomplished  in  a  single 
EDH-UNrr-TASK  operator,  down  to  the  keystroke  level  where  the  operators  were  keystrokes  and  mouse- 
movements. 
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display  for  the  inputting  information  to  the  user  (e.g.,  marked-up  manuscript  for  text 
editing,  sketch  of  a  circuit  for  VLSI  layout).  They  allowed  the  user  to  work  at  her  or  his 
own  pace;  the  user  did  not  have  to  wait  for  critical  information  to  appear,  and  critical 
information  did  not  disappear.  Subtasks  were  not  thrust  upon  the  user  by  a  changing 
environment.  Thus,  these  early  studies  did  not  demonstrate  the  appropriateness  of  the 
GOMS  models  for  interactive  tasks. 

Since  its  introduction,  GOMS  analyses  have  been  used  to  model  many  tasks  in  a  diversity 
of  domains  and  several  extensions  have  been  made  to  the  original  formulation  (Olson  & 
Olson,  1990).  GOMS  models  have  been  demonstrated  within  a  production  system 
architecture  to  model  performance  and  learning  (Kieras  &  Poison,  1985;  Poison  &  Kieras, 
1985;  Kieras  &  Bovair,  1986;  Singley  &  Anderson,  1989).  The  operators  have  been 
brought  down  to  the  level  of  elementary  perceptual,  cognitive,  and  motor  operations 
(Rosenbloom,  1983;  John,  Rosenbloom,  &  Newell,  1985).  This  allows  perceptual, 
cognitive,  and  motor  processes  to  be  expressed  as  occurring  in  parallel  when  the  task 
environment  allows  look-ahead  or  anticipation  of  operations  (John  &  Newell,  1989). 

Errors  attributed  to  working-memory  limitations  have  been  predicted  (Lerch,  Mantei,  & 
Olson,  1989).  Verbal  input  and  output  and  tasks  driven  by  the  environment  have  extended 
the  characteristics  of  GOMS  tasks  (John,  1990).  Task  domains  modelled  in  GOMS  now 
include  graphic  editors  (Ziegler,  Hoppe,  &  Fahnrich,  1986),  spreadsheets  (Olson  &  Nilsen, 

1988) ,  computer  command  abbreviations  (John,  et  al.,  1985;  John  &  Newell,  1987) 
oscilloscopes  (Lee,  Poison,  &  Bailey,  1989),  touch  typing  (John,  1988;  John  &  Newell, 

1989) ,  and  telephone  operator  call  handling  (John,  1990;  Gray,  John,  Stuart,  Lawrence,  & 
Atwood,  1990). 

gQMSv  THE.  MQPEl  HUMAN_EB-QggSSQB..  ANp  SOAR 

Since  GOMS  provides  a  formalism  within  which  to  describe  and  predict  a  range  of  use- 
behaviors,  it  could  be  considered  a  closed  model,  sufficient  in  itself.  In  actuality,  GOMS 
presupposes  an  underlying  general  theory  of  human  cognition,  for  which  it  is  the 
specialization  and  instantiation  of  that  theory  to  specific  task  environments  and  types  of 
humans.  GOMS  was  initially  presented  exactly  this  way  (Card,  et.  al.,  1983).  The 
underlying  cognitive  theory  was  the  Model  Human  Processor  (MHP).  It  comprised  an 
architectural  structure,  composed  of  memories  (long-term  memory,  working  memory  and 
sensory  buffers)  and  processors  (perceptual,  cognitive  and  motor).  The  operation  of  this 
structure  was  not  given  in  full  detail  (as  in  a  computer  architecture).  Instead  a  series  of 
operating  principles  were  given,  such  as  decreasing  speed  of  the  cognitive  processor  with 
uncertainty,  and  the  use  of  problem  spaces  as  an  overall  way  of  organizing  performance. 
The  MHP  was  meant  to  summarize,  circa  the  early  1980's,  what  had  been  learned  in 
cognitive  science  about  the  operation  of  basic  information  processing.  The  version  of 
GOMS  originally  presented  was  entirely  consistent  with  the  MHP.  In  turn,  the  MHP  was 
meant  to  do  more  than  simply  justify  GOMS.  It  was  to  provide  a  basis  for  reasoning  more 
generally  about  users  interacting  with  computers.  (Indeed,  it  made  no  pretense  to  be  a 
general  model  of  all  human  cognition,  but  was  itself  a  specialization  to  the  situations  of 
interest  to  HCI.) 

Tne  MHP  was  far  from  a  complete  cognitive  theory,  even  in  the  HCI  domain  to  which  it 
was  specialized.  The  state  of  die  art  of  cognitive  science  was  simply  not  sufficient  to 
provide  such  a  theory.  Nevertheless,  the  MHP  has  proved  sufficient  to  conceptually 
support  all  of  the  extensions  that  have  been  made  to  GOMS.  The  MHP  was  cast  in  terms 
of  productions  and  recognition-act  cycles;  this  provided  an  alternative  formalism  to  the 
procedural-language  formalism  of  the  original  GOMS,  permitting  the  extension  to  skill 
learning.  The  MHP  defined  enough  internal  memory  structure  for  the  cognitive  processor 
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to  support  the  refinement  of  GOMS  operators  from  the  half  second  level  down  to  the  50 
msec  level.  This  memory  structure  was  also  sufficient  to  support  the  extension  of  GOMS 
to  account  for  some  working  memory  errors.  And,  finally,  the  three-processor  structure  of 
the  Miff  (perceptual,  cognitive  and  motor)  was  sufficient  to  extend  GOM_>  to  continuous 
interaction  tasks,  such  as  typing  and  talking-while-keying  tasks,  where  the  human  operates 
in  a  pipelined,  overlapped  mode. 

Although  it  is  impressive  that  the  MHP  has  been  conceptually  rich  enough  to  support  all 
these  extensions,  the  MHP  remains  a  highly  sketchy  theory.  The  memories  are 
characterized  only  by  a  few  descriptive  parameters  (such  as  half-life)  and  the  principles  of 
operation  are  not  computationally  described  (for  example,  it  is  simply  stated  that  the  power 
law  of  practice  is  obeyed).  It  is  not  literally  possible  to  derive  GOMS  (in  any  of  its 
variants)  from  the  MHP.  Additional  detail  is  always  required,  for  example,  the  particular 
procedural  language  structure  of  the  original  GOMS.  What  remains  true  is  that  detailed 
GOMS  systems  are  further  specifications  of  the  structure  and  principles  of  the  MHP. 
Finally,  unlike  GOMS  itself,  which  has  been  continuously  tested  and  extended  since  it  was 
initially  introduced,  no  such  activity  of  updating,  exercising  or  refinement  has  occurred  for 
the  MHP.  Thus,  good  reasons  exist  to  adopt  a  better  base  for  GOMS  analyses  than  the 
MHP,  if  a  suitable  one  can  be  found.  There  are  some  minimal  conditions  for  any  basic 
psychological  theory  to  support  GOMS,  namely,  that  it  support  the  particular  control 
structure  of  goals,  operators,  methods  and  selections  rules.  Many  cognitive  architectures 
can  do  this,  but  certainly  not  all,  e.g.,  Act*  (Anderson,  1983)  can,  but  in  general 
connectionist  architectures  cannot  (Rumelhart,  et.  al.,  1986).  Furthermore,  the  real 
leverage  in  a  new  base  should  be  the  ability  to  support  further  extensions  of  GOMS  and  its 
integration  with  the  many  other  cognitive  and  perceptual  processes  that  arc  relevant  to 
interacting  with  computers. 

Soar  is  a  recent  attempt  to  provide  an  architecture  for  human  cognition  (Laird,  Newell  & 
Rosenbloom,  1987;  Newell,1990).  It  is  generally  consonant  with  the  MHP,  and  hence 
supports  the  basic  control  mechanisms  of  GOMS.  But,  unlike  the  MHP,  Soar  is  a 
completely  specified  architecture.  That  is,  rather  than  being  given  as  an  abstract 
memory/process  structure  plus  an  set  of  abstract  principles  of  operation,  Soar  is  given  in 
the  fashion  of  a  programmed  computer,  with  data  structures,  memory  accessing 
organization,  and  full  details  of  the  operation  of  the  processors.  Thus,  one  can  specify  the 
contents  of  Soar  memory  structures  for  particular  users  in  particular  task  situations.  These 
contents,  in  effect,  program  Soar  so  that  it  produces  simulations  of  the  behavior  of  the  user 
in  the  task. 

As  succinctly  described  in  (Lewis,  et  al.,  1990)  and  in  more  detail  elsewhere  (Laird,  et.  al., 
1987;  Newell, 1990),  the  Soar  architecture  formulates  all  tasks  in  problem  spaces ,  in  which 
operators  arc  selectively  applied  to  the  current  state  to  attain  desired  states.  Problem 
solving  proceeds  in  a  sequence  of  decision  cycles  that  select  problem  spaces,  states  and 
operators,  resulting  in  the  application  of  the  operator  to  move  to  a  new  state  in  the  space. 
Each  decision  cycle  accumulates  knowledge  from  a  long-term  recognition  memory  (realized 
as  a  production  system).  This  memory  continually  matches  against  working  memory, 
elaborating  the  current  state  and  retrieving  preferences  that  encode  knowledge  about  the 
next  step  to  take.  Access  of  recognition  memory  is  involuntary,  parallel,  and  rapid  (on  the 
order  of  10  msec).  The  decision  cycle  accesses  recognition  memory  repeatedly  until 
quiescence,  when  no  more  knowledge  can  be  brought  to  bear,  then  the  decision  is  made 
about  which  step  to  take  next  Each  decision  cycle  takes  on  the  order  of  100  msec. 

If,  at  quiescence,  the  accumulated  coded  knowledge  in  working  memory  is  insufficient  (or 
conflicting),  so  that  Soar’s  next  step  cannot  be  determined,  then  an  impasse  occurs.  Soar 
responds  to  an  impasse  by  creating  a  subgoal  in  which  a  new  problem  space  can  be  used  to 
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acquire  the  needed  knowledge  (or  resolve  the  conflict).  If,  similarly,  lack  of  knowledge 
prevents  progress  in  this  new  space,  another  impasse  occurs  md  another  subgoal  is  created 
--  and  so  on,  leading  to  an  entire  goal-subgoal  hierarchy.  Once  an  impasse  is  resolved  by 
problem  solving,  the  chunking  mechanism  adds  new  productions  to  the  recognition 
memory  that  encode  the  results  of  the  problem  solving.  Thus,  the  impasse  is  avoided  in  the 
future,  because  these  productions  provide  the  appropriate  knowledge  immediately. 

Soar  interacts  with  the  external  environment  through  perceptual  and  motor  processes, 
which  operate  through  the  working  memory.  Incoming  perceptions  are  added  to  the 
current  state  in  the  top  problem  space  and  motor  commands  are  made  part  of  this  current 
state.  The  interactions  occur  asynchronously  with  the  operation  of  the  cognitive  decision 
cycle. 

Soar  basically  subsumes  the  MHP  and  provides  a  much  better  basis  on  which  to  construct 
GOMS  models.  However,  the  situation  is  not  perfect.  It  remains  an  open  issue  to  show 
that  some  of  the  MHP  operating  principles  expressing  global  psychological  laws  (such  as 
Fitts'  law  and  Hick's  law)  follow  from  the  Soar  architecture.  (Some  global  laws,  such  as 
the  power  law  of  practice,  have  already  been  shown  to  be  characteristic  of  the  architecture.) 
Some  of  the  ways  in  which  Soar  does  not  yet  adequately  subsume  the  MHP  relate  to  the 
perceptual  and  motor  processors,  which  remain  underspecified  aspects  of  the  Soar 
architecture.  These  difficulties  do  not  substantially  affect  taking  GOMS  models  as 
specializations  of  the  Soar  architecture  for  appropriate  users  doing  appropriate  tasks.  In 
fact,  the  basic  GOMS  model  refined  for  50  msec  operators  has  been  derived  from  Soar 
(Newell,  1990,  Chapter  5).  This  extends  easily  to  longer  duration  GOMS  operators,  to  the 
learning  effects  (in  terms  of  Soar  chunking),  and  to  the  three-processor  GOMS  models  of 
continuous  behavior,  although  all  the  details  have  not  yet  been  fully  worked  out. 

Soar  is  a  better  basis  than  the  MHP  on  which  to  construct  GOMS  models  for  reasons  that 
go  beyond  just  reproducing  the  current  GOMS,  even  if  that  can  be  done  more  elegantly. 

The  most  important  is  that  Soar  is  complete  enough  and  cognitively  adequate  enough  so  :.t 
is  being  used  to  model  and  explain  many  diverse  cognitive  phenomena  (Lewis,  et.  al.  1990; 
Newell,  1990).  Examples  include  natural  language  comprehension,  problem  solving, 
immediate  reasoning,  perceptual  search,  strategy  discovery  and  change,  and  the  taking  of 
instructions.  These  tasks  encompass  the  major  dependent  variables  of  interest  in  HCI  - 
time,  errors,  and  learning  rates.  Thus  GOMS  automatically  is  subsumed  within  a  theory 
which  is  being  extended  to  many  other  areas  of  cognition  relevant  to  HCI.  These 
extensions,  even  when  good  psychology,  do  not  always  automatically  extend  GOMS  in 
ways  that  preserve  its  property  of  permitting  engineering  calculations  and  making 
parameter-free  predictions.  But  they  provide  excellent  starting  points  for  such  extensions. 
The  second  reason  is  the  universal  availability  of  simulations  using  Soar,  so  that  even  when 
analytic  calculations  cannot  be  made,  it  is  possible  to  put  all  the  various  parts  of  a  user 
model  together  and  simulate  what  should  happen  (John,  Newell  &  Card,  1990,  is  an 
example  in  the  domain  of  user's  behavior  with  a  browsing  system). 

Thus,  from  now  on  in  this  paper  we  will  work  within  the  Soar  architecture,  taking  it  as  the 
architectural  foundation  upon  which  to  build  GOMS  models. 

THE  TASK 

Nintendo's  Super  Mario  Bros.  3  was  chosen  for  the  HFS  panel  by  Wayne  Gray  because  it 
is  a  highly  interactive  task  domain,  its  popularity  as  a  video  game  was  likely  to  draw  a  large 
audience  to  the  panel,  and  an  expert,  KP,  was  readily  available  and  willing  to  participate. 
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Super  Mario  Bros.  3  is  a  typical  adventure  game  with  treasure  to  collect,  enemies  to  avoid 
or  kill,  and  super  powers  to  acquire  and  use.  The  user  manipulates  the  hero,  Mario, 
through  the  world  by  pressing  buttons  on  a  hand-held  controller  (Figure  1).  The  game  has 
eight  different  worlds  to  traverse,  and  each  world  has  several  levels,  with  the  difficulty  of 
play  increasing  with  the  world  and  level  number. 


XsONTROLLER  OPERATION 


h 


For  the  1  play9r  game  use  controller  1 
For  the  2  player  game  use  controllers  1  and  2 


Controller  1  /  Controller  2 


A  Button 
B  Button 

START  Button 
L-  SELECT  Button 


<Qj  Control  Pad 


Control  Pad 


Up 

*  Mario  can  enter  a  door. 

*  If  you  press  the  A  Button  at  the 
same  time,  Mario  can  jump  up.out 
of  water. 

*  If  you  press  the  A  Button  at  the 
same  time,  Mario  can  enter  some 
upside-down  pipes. 


Down 

*  Mario  can  squat  (except  for  Frog  Mario.) 

*  Mario  can  enter  some  pipes. 

*  When  the  ground  slopes,  Mario  can  slide  down 
it  (except  for  Frog  Mario.) 


Left  and  Right 

*  Mario  can  walk  to  the  left  and  right.  If  you  hold  the  B 
Button  as  you  go  left  or  right,  Mario  will  run. 


Figure  1.  The  hand-held  controller  for  Super  Mario  Bros.  3.  (from  p.  6,  Super 
Mario  Bros.  3.  instruction  booklet ,  reprinted  by  permission  of  Nintendo  of 
America,  Inc.) 


Several  enemies  populate  these  worlds,  among  them  are  Goombas,  who  can  kill  Mario  by 
running  into  him  but  can  be  killed  by  Mario,  ParaGoombas  (flying  Goombas  with  wings), 
Venus  Fire  Traps ,  invincible  plants  who  live  in  pipes  and  throw  deadly  fireballs,  and 
Koopa  Troopers  (Koopas),  turtle-like  creatures  who  retreat  into  their  shells,  which  can  then 
be  lacked  or  thrown  by  Mario  to  defeat  other  enemies  or  break  open  treasure  blocks. 


Treasures  in  these  worlds  include  coins  that  allow  Mario  to  buy  additional  lives,  and 
mushrooms  and  leaves  that  give  Mario  super  powers  when  he  runs  into  them.  Treasures 


can  appear  directly  on  the  screen,  or  they  can  be  hidden  in  blocks  and  only  appear  when  the 

Both  killing  enemies  and  collecting  treasures  give  Marie 


ui — u  —  i _ i — u..  x 

U1VA/XU>  aio  U1UO.WU  uy  niaiiu. 


points. 


The  display  that  appears  to  the  user  at  the  beginning  of  each  game  is  shown  in  Figure  2. 
This  display  shows  Mario,  four  question  blocks,  a  type  of  block  guaranteed  to  contain  a 
treasure  (QB.l  through  QB.4),  a  Goomba  (G.l),  the  level  ground  that  Mario  walks  on 
(Groundl.l),  and  a  scaffold  he  can  climb  on  to  reach  treasures  or  avoid  enemies 
(Scaff2.1). 
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QB.1  QB.'J 


Mario 


Groundl  .1  — 


Scaf2.1 


-Score  Box 


Figure  2.  Start-up  display  for  Super  Mario  Bit's.  3  World  1  Level  1. 


For  simplicity  sake  (for  both  the  Nintendo-naive  panelists  and  the  ease  of  generating  data) 
Gray  chose  the  task  to  be  the  lowest  level  of  difficulty,  World  1  Level  1.  He  asked  the  nine- 
year-old  KP  to  traverse  World  1  Level  1,  collecting  as  many  points  as  possible,  thinking 
aloud  as  he  played.  Gray  videotaped  KP  performing  this  task  to  provide  observed  behavior 
against  which  to  measure  the  predictions  of  the  GOMS  analyses.  The  panelists  were  asked 
to  analyze  the  first  27  seconds  of  play,  in  which  time  KP  collected  nine  treasures,  killed  five 
enemies,  avoided  three  dangers,  added  two  super  powers,  and  flew  off  the  screen  to 
another  part  of  World  1  Level  1.  The  completion  of  World  1  Level  1  took  KP  and 
additional  43  seconds.  Appendix  I  contains  the  transcription  of  KP's  behavior  for  the 
analyzed  segment 

TWQ-QQM3.  ANALYSES 

GOMS  suggests  that  the  goals,  operators,  and  methods  can  be  defined  from  an  objective 
analysis  of  the  task.  Optimal  selection  rules  can  often  also  be  specified  from  a  task 
analysis,  but  experience  with  human  behavior  indicates  that  selection  rules  in  actual  use  are 
specific  to  an  individual  user  and  must  be  inferred  from  his  or  her  behavior.  We  conducted 
the  GOMS  analyses  with  as  little  reference  to  the  expert's  behavior  as  possible,  opting 
instead,  for  taking  the  knowledge  explicit  in  the  instruction  booklet  and  reasoning  about  the 
task  itself.  We  found  that  the  goals,  operators,  and  methods  sufficient  to  play  the  game 
could  be  inferred  from  the  instruction  booklet  and  task  analysis.  Selection  rules  were 
determined  through  reference  to  the  expert's  behavior. 

The  analyses  were  carried  out  at  two  levels:  the  function-level  where  operators  are  at  the 
level  of  gathering  an  item  that  gives  points  or  killing  an  enemy,  and  the  keystroke-level , 
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where  operators  are  at  the  level  of  individual  finger  movements  on  the  game's  control  panel. 
Each  level  is  considered  a  separate  GOMS  analysis,  as  in  Chapter  5  of  The  Psychology  of 
Human-Computer  Interaction  (Card,  et.  al.  1983)  where  nine  models  at  four  different  levels 
were  compared.  We  will  present  these  analyses  in  parallel,  first  describing  the  goals, 
operators,  methods  and  selection  rules  for  each,  then  describing  the  process  of  applying  this 
information  to  the  task  of  playing  Super  Mario  Bros.  3. 

Goals,  operators,  methods,  and  selection  rules  of  the  task  domain 

The  instruction  booklet  is  the  primary  source  of  knowledge  for  this  GOMS  analysis.  It 
provides  the  overall  goals  of  the  game:  clear  the  level  (in  order  to  go  on  to  the  next  level), 
increase  your  standing  by  accumulating  points,  coins,  cards,  and  extra  lives  (these  are  the 
things  measured  in  the  score  box  at  the  bottom  of  the  screen),  and  avoid  death  due  to  being 
touched  by  an  enemy,  falling  into  a  hole  or  fire,  or  running  out  of  time. 

These  goals  can  be  accomplished  by  ■perlommg  function-level  operators  (FLOs),  also 
inferred  from  the  instruction  booklet.  Figure  3  lists  the  goals  of  the  game,  the  FLOs  that  act 
to  fulfill  those  goals  in  the  segment  of  behavior  analyzed,  and  the  passages  in  the  instruction 
booklet  from  which  the  FLOs  were  inferred.  In  sendee  of  clearing  the  level,  the  FLOs  are 
move-towards-end  (in  effect,  moving  to  the  right)  and  touch-goal,  where  the  goal  is  a  white 
box  in  the  middle  of  a  black  screen  at  the  end  of  the  level  (Mario  must  jump  up  to  touch  it). 
To  increase  standing,  the  FLOs  are  gather-item,  search-in-block,  and  attack-enemy. 
Gather-item  collects  items  on  the  screen  that  give  points  or  money.  Search-in-block  hits 
blocks  to  see  if  something  valuable  comes  out,  which  could  then  be  gathered.  Points  can 
also  be  gotten  with  the  attack-enemy  FLO,  although  there  is  some  risk  involved;  an  attack 
could  fail  and  the  enemy  could  kill  Mario,  violating  the  goal  to  avoid  death.  To  fulfill  the 
goal  to  avoid  death,  the  FLO  is  avoid-danger,  where  a  danger  can  be  an  enemy,  a  hole  in 
the  ground  (if  Mario  falls  in,  he  dies),  or  fire.  There  is  no  FLO  that  prevents  Mario  from 
running  out  of  time.  However,  a  consideration  of  time  is  inherent  in  the  selection  among 
the  other  FLOs. 

FLOs  are  realized  through  keystroke-level  operators  (KLOs).3  The  KLOs  are  the  hand 
movements  necessary  to  manipulate  Mario,  and  involve  hitting  the  six  buttons  on  the 
controller  (Figure  1).  The  four  buttons  in  a  cross  configuration  move  Mario  to  the  right, 
left,  up,  and  down4  and  we  call  the  KLOs  press-right,  press-left,  press-up,  and  press- 
down,  respectively.  The  A-button  makes  Mario  jump,  and  when  he  has  a  tail,  pressing  the 
A-button  repeatedly  makes  Mario  fly.  Holding  the  B-button  down  allows  Mario  to  pick  up 
things  that  lie  next  to  him,  and  to  accelerate  when  used  in  conjunction  with  the  right-  or  left- 
buttons.  Releasing  the  B-button  causes  Mario  to  drop  whatever  he  is  carrying. 

These  two  levels  of  operators  have  a  relationship  described  by  Card,  et  al.  (1983,  Chapter 
5)  as  being  formed  by  splitting  operators.  That  is,  the  FLOs  are  split  into  the  hand 


3 1  use  the  terms  goals,  function-level  operators  and  keystroke-level  operators  to  distinguish  between  three 
different  levels  in  the  typical  GOMS  goal  hierarchy.  This  distinction  is  drawn  to  make  the  different  levels 
easy  to  write  about;  it  docs  not  reflect  a  theoretical  distinction  between  goals  and  operators.  As  Kicras 
(1988)  observes  for  GOMS  analyses  in  general,  "this  distinction  is  intuitively-based,  and  it  is  also  relative; 
it  depends  on  the  level  of  analysis." 

4  Note  that  pressing  the  up  button  does  not  make  Mario  jump  up,  rather  it  allows  him  to  move  up  in 
certain  situations:  to  go  up  a  pipe  with  its  opening  over  his  head  (together  with  the  A-button)  or  pop  out  of 
water  (together  with  the  A  button)  or  to  go  through  a  door.  None  of  these  maneuvers  are  necessary  in 
segment  of  behavior  analyzed  here.  Likewise,  the  down  button  makes  Mario  squat,  go  down  a  pipe  when 
he  is  standing  on  its  opening,  or  to  slide  down  a  hill.  Again,  none  of  these  maneuvers  are  used  in  this 
segment  of  behavior. 
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W.>S( 


GOAL: 

PARAMETERS 

INSTRUCTION  BOOKLET  SOURCE 

PAGE 

FUNCTION-LEVEL 

OPERATOR 

Clear-level: 

"...touch  the  gOe.;  to...clear  the  level" 

19 

Move-towaids-end 

"At  the  end  of  each  action  scene..." 

19 

Touch-goal 

"...touch  the  goal  to...c!ecr  the  level" 

19 

Increase-standing: 

no.of  M?.rios  remaining 
score, number  of  coins, 
cards  taken 

Score  box  at  bottom  of  screen 

16 

Search-in-block 

Question  blocks, 
other  blocks 

"Hit  blocks...A  useful  item  might  pop  outl" 

11 

Gather-tern 

Coin,  Starman,  Super  Leaf, 

"Gain  more  power  by  gathering  items" 

18 

1-Up  Mushroom,  Super 
Mushroom,  Fire  Flower 

Attack-enemy 

Points  ?;  a  given  for  attacking  enemies 

14 

Goomba 

Attacks  pictured 

12,14 

Koopa 

"After  >ou  have  jumped  on  a  Koopa..." 

8 

Para-Goo  mba 

"Once  you  jump  on  it..." 

35 

Avold-death: 

enemy,  hole,  fire,  time-out 

"BEWARE!  THE  FOLLOWING  ARE 
DEADLY" 

20 

Avoicklanger 

enemy,  hole,  fire 

mentioned  in  deadly  category 

20 

Figure  3.  Goals  and  FLOs  of  the  anlayzed  segment  of  Super  Mario  Bros.  3  World  1  Level  1 


movements  necessary  to  accomplish  them,  defining  the  KLOs.  KLOs  are  at  a  lower  grain- 
size,  and  can  be  combined  in  different  ways  to  accomplish  many  different  FLOs.  As  in 
Card,  et  al.,  determining  the  sequence  of  opera  ors  at  either  of  these  levels  constitutes  a 
single  GOMS  analysis.  Thus,  determining  both  the  sequence  of  FLOs  and  the  sequence  of 
KLOs  is  considered  to  be  two  GOMS  analyses.  However,  since  KLOs  are  simply  split 
FLOs,  the  analysis  process  is  to  determine  the  appropriate  FLO  and  then  determine  the 
KLOs  that  accomplish  it;  the  analyses  thus  proceed  in  parallel. 

The  instruction  booklet  provides  methods ,5  or  sequences  of  KLOs,  for  accomplishing  some 
FLOs  (examples  of  methods  from  the  instruction  booklet  are  shown  in  Figure  4).  Figure  5 
lists  the  functional  operators  necessary  to  produce  the  observed  behavior,  the  name  of  the 
met  ids  provided  by  the  instruction  manual  for  accomplishing  those  FLOs,  and  the  moves 
that  make  up  the  methods,  and  the  KLOs  that  produce  those  moves.  The  exact  KLOs 
making  up  a  method  depend  on  the  relative  position  of  Mario  and  the  objects  on  the  screen, 
so  the  keystrokes  given  are  illustrative  (e.g.,  a  block  may  be  to  the  right  or  left  of  I/Iario,  so 
the  KLO  producing  appropriate  movement  would  be  press-right-button  or  press-left-button, 
respectively). 


5  The  instruction  booklet  uses  the  term  techniques  to  describe  what  GOMS  calls  methods. 
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NEW  TECHNIQUES! 


Holding  a  shell 

*s®] 


Running  with  a  shell 


o®@i 


(Holding  the  E)  Button)  (Holding  the  B  Button) 


Kicking  the  shell 

Q(D® 

(Releasing  the  B  Button) 


£ 


Breaking  a  block 


When  Mario  has  a  tail 


Accelerating 


Power  Meter 
going  up 


More  acceleration 


Meter  lull,  (P)  starting 
to  flash 


Take  off 

Qd)® 

Press  the  A  Button 
repeatedly 


Mario  can  only  fly  for 
a  short  time. 


0®<8> 


Figure  4.  Methods  suggested  by  the  instruction  booklet  (from  p.  11.  Super  Mario 
Bros.  3.  instruction  booklet,  reprinted  by  permission  of  Nintendo  of  A’  lerica,  Inc.) 


More  than  one  method  exists  to  accomplish  each  FLO  used  in  the  segment  of  the  game 
studied  (except  for  gather-item).  We  determined  selection  rules  for  these  methods  by 
analyzing  two  sources  of  information:  the  mechanics  of  the  game  itself  (the  necessary 
conditions  for  the  methods  to  be  applied),  and  the  expert  behavior  observed  in  the 
videotape. 

For  the  search-in-block  FLO,  the  necessary  conditions  for  the  hit-from-bottom  method  are 
that  the  block  be  above  Mario  and  that  there  be  a  clear  path  for  him  to  jump  up  and  hit  it 
For  the  hit-with-shell  method,  the  necessary  conditions  are  that  there  be  a  Koopa  Trooper 
available  to  supply  a  shell  and  that  ihere  be  a  vantage  point  from  which  the  shell  can  be 
thrown  to  hit  the  block.  For  the  hit-with-tail  method,  Mario  must  possess  a  tail  and  the 
block  must  be  vulnerable  to  tail-attack.  In  this  segment  of  the  game,  none  of  the  blocks  are 
vulnerable  to  tail-attack,  so  this  method  is  never  observed.  Because  the  hit-from-bottom- 
method  has  fewer  steps  it  takes  less  time  than  the  hit-with-shell  methods;  because  it  does 
not  involve  an  enemy,  it  has  less  risk  associated  with  it.  Therefore,  the  selection  rule  we 
inferred  is  that,  if  the  necessary  conditions  for  the  hit-from-bottom  method  exist,  then  the 
hit-from-bottom  method  will  be  selected.  If  not,  then  the  hit-with-shell  method  will  be 
selected.  In  this  segment  of  behavior  it  is  the  case  that  if  the  necessary  conditions  for  the 
hit-from-bottom  method  do  not  exist,  the  necessary  conditions  for  the  hit-with-shell  method 
exist,  so  this  selection  rule  works.  Reference  to  the  expert’s  behavior  did  not  contradict 
this  inferred  selection  rule. 


FLOs  USED 

IN  OBSERVED 
SEGMENT 

POSSIBLE 

METHODS 

CONDITIONS  OF 
SELECTION 
RULES 

MOVES  IN 
METHOD 

ILLUSTRATIVE 

KLOs  TO 

ACCOMPLISH  MOVES 

search-in-block 

hit  from  bottom 

block  is  above  Mario 
path  below  block  is  clear 

move  undl  under  block 
jump 

press-right-button 

press-A-button 

hit  with  shell 

Koopa  Trooper  available 
vantage  point  available 

move  to  Kooppa 
immobilize  Koopa 
pick  up  shell 
move  to  block 

throw  shell  at  block 

press-right-button 

press-A-button 

hold-B-button 

press-right-button 

release-B-button 

hit  with  tail 

Mario  has  a  tail 

block  breakable  via  tail 

move  to  block 

hit  block  with  tail 

press-right-button 

press-B-button 

gather-item 

move  into 

move  to  item 

press-right-button 

attack-enemy 

stomp  on 

always  applicable 
always  selected 

move  to  enemy 
jump  on  enemy 

press-right-button 

press-A-button 

tail  attack 

Mario  has  a  tail 
(not  used  by  this  expert) 

move  to  enemy 
hit  with  tail 

press-right-button 

press-B-button 

avoid-danger 

get  out  of  way 

a  clear  escape  route  exists 
does  not  impede  progress 
toward  end  of  level 

move  away 

press-right-button 

jump  over 

clear  area  above  Mario's 

head 

clear  spot  to  jump  to 
"get  out  of  way"  impedes 
progress  toward  end 

jump  over 

press-right-button 

press-A-button 

fly  over 

Mario  has  a  tail 
clear  runway 
clear  take-off  airspace 
only  when  there 
are  no  other  options 

walk  back 

turn  around 

accelerate 

take-off  and  fly 

press-left-button 

press-right-button 

press-right-button 

hold-B-button 

release-B-button 
press-A-button  (repeatedly) 

-  move-toward-end 

walk 

always  selected 

walk  to  the  right 

press-right-button 

run 

not  used 

run  to  the  right 

press-right-button 

hold-B-button 


Figure  5  -  ELOs,  methods,  selection  rules,  and  illustrative  KLOs  for  the 
analyzed  segment  of  the  experts  behavior. 
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For  the  attack-enemy  FLO,  the  necessary  condition  for  the  tail-attack  method  is  that  Mario 
have  a  tail.  When  Mario  does  not  have  a  tail,  then  there  is  no  selection  rule  because  only  the 
stomp-on  method  is  applicable,  and  this  is  the  situation  for  the  first  enemy  attacked.  When 
Mario  has  a  tail,  the  selection  rule  may  be  determined  by  analyzing  the  rewards  and 
penalties  associated  with  each  method.  For  instance,  the  stomp-on  method  seems  to  require 
more  precise  timing  of  action  and  has  more  risk  of  dying  for  a  novice  than  the  tail  attack.  It 
is  assumed  that  an  expert  can  execute  whichever  maneuver  he  or  she  intends,  so  the  risk 
component  probably  does  not  enter  into  the  determination  of  the  expert's  selection  rule.  In 
the  case  of  killing  a  ParaGoomba,  a  tail  attack  gets  only  100  points  whereas  the  two 
successive  jumps  required  to  kill  the  enemy  using  the  stomp-on  method  get  100  and  200 
points  respectively,  giving  200  more  points  than  the  tail-attack  method.  This  must  be 
traded-off  against  the  extra  time  it  takes  to  kill  a  ParaGoomba  with  two  jumps,  about  800 
msec.  For  World-1  Level-1,  the  average  rate  of  point  collection  is  150  points  per  second, 
so  the  extra  800  msec  to  kill  the  ParaGoomba  by  jumping  is  well  worth  the  200  extra  points 
that  method  produces.  Therefore,  the  selection  rule  for  attack-enemy,  when  the  enemy  is  a 
ParaGoomba  is  to  use  the  stomp-on  method  exclusively.  In  the  case  of  killing  a  Goomba, 
the  selection  rule  is  not  so  clear,  the  tail-attack  and  stomp-on  method  seem  to  have  the  same 
risk  (small)  and  the  same  reward  (100  points).  However,  the  tail-attack  method  is  described 
in  the  instruction  manual  as  being  a  new  technique  for  Super  Mario  Bros.  3.  Since  our 
expert  was  an  expert  in  Super  Mario  Bros.  1  and  2  before  this  game,  it  is  reasonable  to 
assume  a  bias  towards  familiar  methods,  again  giving  a  selection  rule  that  uses  stomp-on 
exclusively.6  These  selection  rules  were  not  contradicted  by  the  expert's  behavior  in  this 
segment. 

For  the  avoid-danger  FLO,  the  necessary  condition  for  the  get-out-of-the-way  method  is 
that  there  be  a  direction  of  escape,  right,  left,  up  or  down.  The  necessary  condition  for  the 
jump-over  method  is  that  there  be  clear  space  over  Mario's  head  so  he  can  jump.  The 
necessary  conditions  for  the  fly-over  method  are  that  Mario  has  a  tail,  there  is  a  clear  run- 
.vay  to  get  up  to  speed  and  nothing  overhead  to  block  the  take-off.  The  fly-over  method 
takes  so  many  more  KLOs  (to  back  up,  accelerate  and  flap),  that  it  should  only  be  selected 
when  there  are  no  other  options;  this  situation  does  not  occur  in  the  observed  segment.  The 
jump-over  method  takes  one  more  KLO  than  the  get-out-of-way  method.  However,  if  the 
get-out-of-way  method  is  always  taken  when  a  danger  is  present,  it  may  prevent  forward 
progress  in  the  game.  For  example,  a  pipe  with  a  plant  in  it  throwing  fireballs  at  Mario  may 
be  to  the  right  of  Mario.  The  get-out-of-way  method  may  send  Mario  running  to  the  left 
until  he  is  out  of  range  of  the  fireballs.  However,  Mario  would  never  progress  beyond  the 
fire-throwing  plant  with  this  selection  rale  in  force,  violating  the  clear-level  goal  and  making 
Mario  run  out  of  time,  thereby  violating  its  own  avoid-death  goal.  Therefore,  the  selection 
rule  we  infer  is  if  the  get-out-of-way  method  does  not  impede  forward  progress  toward  the 
end  of  the  level,  then  use  it,  otherwise  use  the  jump-over  method  for  avoiding  danger.  The 
expert's  behavior  does  not  contradict  this  selection  rule. 

For  the  move-towards-end  FLO,  the  methods  are  walk  and  run.  The  videotape  did  not  to 
show  clearly  when  the  B-button  was  held  down  and  when  it  was  not  Also,  the  movement 
of  Mario  on  the  screen  does  not  necessarily  distinguish  between  these  two  methods  without 
deeper  knowledge  of  the  mechanics  of  Mario's  world  (i.e.  momentum).  Therefore,  we  will 
not  distinguish  between  these  methods  in  our  analyses,  and  just  assume  that  move-towards- 
end  is  implemented  by  a  press-right-button  KLO. 


6  In  the  case  of  immobilizing  a  Koopa  Trooper,  the  tail-attack  method  knocks  the  Koopa  of  the  screen,  so 
its  shell  is  not  available  for  use.  Thus  the  attack  of  a  Koopa  Trooper  embedded  in  the  hit-with-shell  method 
of  the  search-in-block  FLO  never  uses  the  tail-attack  method.  This  is  one  of  the  reasons  why  the  attack  of 
the  Koopa  Trooper  embedded  in  the  search-in-block  FLO  is  not  treated  as  a  attack-enemy  FLO. 
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The  GOMS  analysis  process 

Given  these  goals,  FLOs,  KLOs,  methods  and  selection  mles,  analyzing  the  game  involves 
perceiving  the  elements  on  the  display  and  selecting  the  appropriate  operators.  First,  a  FLO 
is  selected,  then  it  is  implemented  by  a  series  of  KLOs,  as  described  in  Figure  6.  The 
particular  sequence  of  KLOs  depend  on  the  method  used  to  implement  the  FLO  and  the 
relative  position  of  Mario  and  the  elemenis  on  the  screen.  The  selection  rules  used  to 
choose  the  appropriate  method  were  described  in  the  previous  section  and  also  appear  in 
Figure  6.  In  the  segment  of  behavior  analyzed,  these  selection  rules  always  result  in  a 
unique  method,  so  there  is  no  conflict  resolution  necessaiy  for  method  selection. 

Therefore,  at  this  stage  in  the  analysis,  we  can  uniquely  determine  the  sequence  of  KLOs 
necessary  to  accomplish  each  FLO.  This  process  of  selecting  FLOs  constitutes  the 
function-level  GOMS  analysis;  that  of  selecting  KLOs  constitutes  the  keystroke-level 
GOMS  analysis. 

search-in-biock(x) 

LI  is  acceptable  when  block(x)  is  on  the  screen; 

L2  has  preference  over  any  other  operator  (best); 

L3  if  there  is  no  selected  search-in-block(x)  operator  whose  application  is  incomplete, 

L4  and  block(x)  is  closer  to  Mario  than  block(y), 

L5  then  search-in-block(x)  is  better  than  search-in-block(y). 

gather-item(x): 

is  acceptable  when  item(x)  is  on  the  screen; 
has  preference  over  any  other  operator  (best); 

if  there  is  no  selected  gather-item(x)  operator  whose  application  is  incomplete, 
and  item(x)  is  closer  to  Mario  than  item(y), 
then  gather-item(x)  is  better  than  gather-item(y); 
if  item(x)  is  moving  and  item(y)  is  not  moving, 
then  gather-item(x)  is  better  than  gather-item(y); 
if  item(x)  is  moving  and  block(x)  is  on  the  screen, 
then  gather-item(x)  is  better  than  search-in-block(x). 

attack-enemy(x): 

is  acceptable  when  enemy(x)  is  on  the  screen  and  enemy(x)  is  killable; 
if  enemy(x)  is  an  immediate  threat, 

then  attack-enemy(x)  is  better  than  search-in-block,  gather-item,  avoid-danger,  or  move-towards-end; 
if  enemy(x)  is  not  an  immediate  threat, 

then  attack-enemy(x)  is  worse  than  move-towards-end. 

avoid-danger(x): 

is  acceptable  when  danger(x)  is  on  the  screen  and  danger(x)  is  invincible; 
if  danger(x)  is  an  immediate  threat, 

then  avoid-danger(x)  is  better  than  search-in-block,  gather-item,  attack-enemy,  or  move-towards-end; 
if  danger(x)  is  not  an  immediate  threat, 

then  avoid-danger(x)  is  worse  than  move-towards-end. 

move-towards-end: 
is  always  acceptable; 

every  other  operator  has  preference  over  move-towards-end  (worst). 


Figure  6.  Preferences  for  the  proposal  and  selection  of  FLOs. 


14 


These  analyses  use  a  hand-simulation  of  what  would  happen  if  the  GOMS  models  could 
interface  directly  with  the  game.  That  is,  the  analyst  looks  at  the  display,  notes  the  elements 
visible,  and  uses  the  information  in  the  preceding  section  to  select  an  FLO  and  appropriate 
KLOs.  Then  the  analyst  presses  the  button  corresponding  to  the  next  KLO  to  be 
performed.  The  button  is  only  pressed  until  it  has  accomplished  its  function  or  until  a 
display  change  occurs  that  creates  the  opportunity  for  a  new  FLO  to  be  selected.  For 
example,  the  hit-from-bottom  method  for  the  search-in-block  FLO  requires  the  analyst  to 
move  Mario  horizontally  until  he  is  under  the  block  to  be  searched.  If  the  block  is  to  the 
right  of  Mario,  the  analyst  presses  the  A  button  until  either  Mario  is  under  the  block  or 
another  element  appears  on  the  display.  This  would  be  counted  as  one  press-right-button 
KLO  in  the  analysis.  If  no  additional  elements  appear  on  the  display  before  Mario  reaches 
the  block,  the  analyst  releases  the  right-button  (stopping  Mario)  then  presses  the  A-button 
momentarily  to  make  him  jump  straight  up  to  break  the  block  and  land  back  below  the 
block.  The  new  display  (now  without  the  block  just  searched)  is  then  analyzed  to  determine 
the  next  FLO  and  KLO,  and  the  analysis  cycle  repeats. 

In  a  Soar/GOMS  model  each  FLO  is  proposed  when  certain  elements  are  visible  on  the 
display.  Each  FLO  has  a  set  of  preferences  associated  with  it;  proposing  an  operator  is 
equivalent  to  making  the  preference  for  that  operator  acceptable.  Other  preferences  may 
serve  to  resolve  conflicts  between  competing  FLOs.  The  condiuons  for  proposal  and  the 
associated  preferences  are  an  integral  part  of  the  GOMS  analyses,  and,  in  this  analysis, 
have  been  determined  through  a  series  of  common-sense  judgments  about  the  task, 
described  below. 

Rules  for  proposing  FLOs  and  preferences  between  competing  operators  are  shown  in 
Figure  6;  a  detailed  explanation  of  the  meaning  of  the  preferences  for  the  search-in-block 
FLO  follows  to  help  interpret  that  figure. 

The  search-in-block  FLO  is  proposed  (has  an  acceptable  preference)  any  time  there  is  an 
unexplored  block  on  the  display.  In  general,  a  search-in-block  FLO  is  the  best  thing  to  do 
because  it  will  often  result  in  finding  valuable  items  which  give  super  powers,  points,  or 
money;  this  is  stated  in  the  instruction  booklet,  "Hit  blocks... A  useful  item  might  pop  out!". 
Thus,  search-in-block  is  always  given  a  best  preference.7  More  than  one  unexplored  block 
may  be  on  the  display  at  one  time,  so  several  search-in-block  FLOs  may  be  proposed,  each 
evoked  by  one  of  the  unexplored  blocks,  and  each  with  a  best  preference.  To  resolve  the 
conflict  that  arises  from  the  existence  of  several  best  operators,  other  preferences  are  added 
to  the  operators,  again  based  on  task  analysis,  as  follows. 

The  overall  goal  of  attaining  the  most  points  possible  dictates  that  the  player  be  efficient  in 
his  or  her  actions,  not  only  because  running  out  of  time  kills  Mario,  but  because  any 
remaining  time  is  converted  into  points  at  the  end  of  the  level.  Thus,  the  block  closest  to 
Mario  should  be  searched  first  This  is  produced  by  making  the  search-in-block  FLO  for 
the  closest  block  better  than  any  of  the  other  proposed  search-in-block  FLOs  (L4  &  L5  in 
Figure  6). 

Occasionally,  in  the  course  of  implementing  a  search-in-block  FLO,  Mario  must  move 
closer  to  an  unexplored  block  (call  it  B)  other  than  the  current  target  (A).  As  they  stand  in 
the  preceding  paragraph,  the  preferences  for  search-in-block  FLOs  would  change  so  that 


7  A  best  preference  means  that  this  operator  is  best  if  there  are  no  other  preferences  that  supersede  best 
However,  other  preferences  can  be  in  place  that  would  make  another  operator  better  than  the  operator  that  is 
best,  or  reject  the  best  operator,  or  create  a  tie  with  other  best  operators.  For  more  details  on  the  semantics 
of  Soar  preferences,  see  Soar  5  Manual. 
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seaich-in-block(B)  would  be  better  than  search-in-block(A)  as  soon  as  Mario  moves  closer 
to  block-B.  This  might  be  a  reasonable  strategy  in  many  cases,  exploring  the  closest  block 
and  going  back  to  missed  blocks  later.  However,  it  is  easy  to  imagine  arrangements  of 
blocks  that  would  cause  the  player  to  skip  blocks  and  never  get  back  to  them  because  the 
presence  of  other  blocks  would  continue  to  direct  the  preferences  away  from  the  slapped 
blocks.  Thus,  to  get  as  many  points  as  possible,  the  preferences  must  be  adapted  to 
prevent  such  target-switching  when  searching  blocks.  So  the  closer  block  can  be  better 
only  when  there  is  no  existing  search-in-block  FLO  that  has  not  yet  been  accomplished  (L3 
in  Figure  6).  Reasoning  similar  to  that  presented  here  for  the  search-in-block  FLO  is  used 
to  construct  the  complete  list  of  preferences  shown  in  Figure  6. 

To  illustrate  the  course  of  an  analysis,  refer  to  Figure  2,  the  first  display  of  the  game,  and 
Figure  7,  a  chart  describing  the  performance  of  the  model.  (Complete  charts  for  the 
analyzed  segment  appear  in  Appendix  H)  In  the  display,  Mario  is  on  the  extreme  left,  on 
the  ground.  There  are  four  unexplored  question  blocks  visible  (QB.l  through  QB.4)  and 
an  enemy  (called  a  Goomba  and  labeled  G.l).  QB.l  and  QB.2  are  within  reach  of  a  jump, 
whereas  QB.3  and  QB.4  are  too  high  and  Mario  must  jump  onto  the  scaffold  below  QB.4 
to  be  able  to  reach  them.  This  information  is  appears  in  Figure  7,  in  the  boxes  labelled 
OBJECTS  SEEN  and  POSITION. 

Each  QB  and  the  Goomba  cause  an  FLO  to  be  proposed,  according  to  the  preferences  in 
Figure  6,  and  recorded  in  Figure  7's  box  PROPOSED  FLO.  The  position  of  the  QBs  and 
Goomba  cause  preferences  to  be  installed,  shown  in  Figure  7's  PREFERENCES  box.  In 
this  case,  search-in-block(QB.l)  has  a  best  preference  and  is  better  than  search-in- 
block(QB.2),  search-in-block(QB.3),  and  search-in-block(QB.4)  because  it  is  closes1-  to 
Mario.  Since  the  Goomba  is  on  the  far  right  of  the  screen,  it  is  not  an  immediate  threat  to 
Mario  and  the  preferences  make  attack-enemy(G.l)  worse  than  the  default  FLO,  move- 
towards-end  (always  acceptable,  but  worst).  Given  the  semantics  of  Soar  preferences, 
search-for-block(QB.  1)  is  selected  as  the  FLO  to  implement;  this  appears  in  the  FLO 
SELECTED  box. 

The  selected  FLO  is  then  implemented  with  a  method  comprised  of  several  KLOs.  The 
appropriate  method  is  determined  with  the  selection  rules  of  Figure  5  and  appears  in  the 
METHOD  box  in  Figure  7.  The  moves  that  Mario  has  to  make  to  accomplish  that  method 
are  listed  in  the  NECESSARY  MOVES  box.  These  moves  are  achieved  by  specific  KLOs 
listed  in  the  KLO  box.  As  previously  described,  these  KLOs  are  performed  by  hand  by  the 
analyst  to  determine  their  effects  on  the  game's  display. 

Since  this  game  is  highly  interactive,  the  situation  may  change  before  all  the  KLOs 
necessary  to  implement  a  specific  FLO  can  be  performed.  This  happens  at  the  very 
beginning  of  the  game;  as  the  player  presses  the  right-button  to  move  Mario  under  QB.l, 
the  Goomba  runs  to  the  left  and  becomes  an  immediate  threat  before  Mario  reaches  QB.l. 
When  the  Goomba  becomes  an  immediate  threat,  the  preferences  for  FLOs  change  and  must 
be  reassessed.  Thus,  the  press-right-button  KLO  is  performed,  but  the  press-A-button 
KLO  (which  would  make  Mario  jump  into  the  QB)  is  not  performed  before  the  situation 
changes  sufficiently  to  interrupt  the  implementation  of  search-in-block(QB.l).  Figure  7 
records  inis  information  in  the  KLO  PERFORMED  box  and  indicates  which  KLO  triggers 
the  next  display  that  causes  a  change  in  preferences  to  occur. 

Appendix  II  contains  charts  like  the  one  in  Figure  7  that  follow  from  the  FLOs,  methods, 
selection  rules,  preferences,  and  KLOs  described  above.  To  create  those  charts,  we  started 
with  the  initial  display  of  the  game,  and  moved  Mario  one  KLO  at  a  time  until  the  elements 
on  the  screen  changed  sufficiently  to  require  reassessment  of  preferences.  That  is,  at  the 
initial  display,  we  pressed  the  right-button  until  either  Mario  was  under  QB.l  (where  the 


Display.  0  (Start-up  screen)  Mario  starts  at  far  left  and  Goomba  is  at  far  right  moving  left. 

Mario  is  trying  to  get  the  first  question  block. 
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method  for  implementing  search-in  block(QB .  1 )  would  require  a  press-A-button  KLO)  or 
something  else  happened  to  change  the  preferences.  In  this  case,  the  Goomba  became  an 
immediate  threat  before  Mario  was  under  QB.l,  so  another  chart  was  created  (Display.  1  in 
Appendix  II). 

To  see  the  sequence  of  FLCs  predicted  by  the  model,  read  the  entry  in  each  chart's  FLO 
SELECTED  box.  To  see  the  sequence  of  KLOs  predicted  by  the  model,  read  the  entries  in 
each  chart's  KLO  box  that  are  marked  performed  in  the  KLO  PERFORMED  box. 


We  compared  the  model’s  predictions  to  the  observed  performance  of  the  expert,  KP,  using 
the  charts  in  Appendix  H  and  the  transcription  of  KP's  performance  in  Appendix  I.  KP's 
verbal  protocol  lagged  his  performance  and  in  some  cases  later  in  the  game  he  had  to  stop 
speaking  altogether  to  continue  to  play  the  game.  We  believe  this  indicates  that  the  verbal 
report  of  his  goals  was  treated  as  a  secondary  task,  dropping  out  when  the  primary  task  of 
playing  the  game  was  too  difficult,  and  may  not  be  complete.  Therefore,  our  comparison 
uses  only  what  KP  makes  Mario  do  rather  than  what  KP  says  he  is  making  Mario  do. 

Appendix  HI  contains  charts  that  record  the  comparison;  an  example  for  the  first  screen 
display  appears  in  Figure  8.  At  the  top  of  the  chart,  Display.x  corresponds  to  the  Display.x 
in  the  model  charts  (Appendix  II).8 

In  the  comparison  charts,  the  top  row  of  boxes,  OBJECTS  SEEN  and  POSITION,  contains 
the  information  about  what  is  on  the  screen  at  this  point  in  the  videotaped  game  and 
correspond  closely  to  the  same  boxes  in  model  charts.  The  correspondence  between  expert 
and  model  is  not  exact,  because  the  model  and  the  expert  may  not  be  in  exactly  the  same 
position  at  every  display  change.  The  second  row  of  boxes  list  the  OBSERVED 
BEHAVIOR,  the  INFERED  FUNCTION  of  that  behavior,  and  INFERED  FINGER 
ACTION.  The  inferred  function  is  ambiguous  if  more  than  one  function  could  be  served  by 
that  behavior  (e.g.  moving  to  the  right  could  move  Mario  towards  the  end  of  the  level,  or  it 
could  bring  him  under  a  block  so  he  can  search  it),  and  listed  as  a  particular  function  if  only 
one  function  is  obviously  served  (e.g.  jumping  into  a  QB  unambiguously  serves  to  search 
that  QB).  The  finger  action  was  inferred  from  the  behavior  because  it  was  not  directly 
observable  from  the  videotape,  but  most  movements  of  Mario  can  only  be  produced  by 
specific  button  presses,  so  the  inference  is  uncontroversial.  The  third  row  of  boxes  lists 
the  FLOs  that  would  be  consistent  with  the  observed  behavior,  the  specific  FLO  predicted 
by  the  model,  and  the  KLO  that  is  predicted  by  the  model.  The  FLO  predicted  by  the  model 
can  be  compared  directly  to  the  inferred  function  of  the  observed  behavior  in  the  box  above 
it.  The  KLOs  can  be  compared  directly  to  the  inferred  finger  actions  in  the  box  above  it 

Figures  9  and  10  graphically  present  the  comparisons  between  the  observed  expert  behavior 
and  the  model’s  predictions.  Figure  9  shows  the  comparison  at  the  function-level  and 
Figure  10  shows  the  comparison  at  the  keystroke-level.  In  both  figures,  the  operator  names 
are  listed  along  the  vertical  axis.  The  horizontal  axis  is  labelled  with  the  display  number 


8  There  are  more  charts  for  the  model  predictions  than  for  the  expert  comparisons  because  the  expert  moves 
in  a  fluid,  continuous  motion  while  the  model  simulates  Mario  walking  one  step  at  a  time,  stopping  before 
and  after  jumps.  Some  of  the  display  changes  occur  only  when  Marie  i'  moved  beyond  a  specific  place  in 
the  world.  The  model’s  incremental  movement  produces  some  displays  where  the  next  elements  has  not  yet 
appeared  and  the  only  applicable  FLO  is  move-toward-end.  In  contrast,  the  expert's  fluid  movement  moves 
Mario  beyond  the  triggering  point  in  service  of  the  previous  function,  so  the  displays  where  nothing  new 
presents  itself  never  occur. 
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Display.  0  (Start-up  screen ) 


Mario  starts  at  far  left 

Gcomba  is  at  far  right  moving  left 

Mario  is  trying  to  get  the  first  question  block. 


State  of  Small  Mario 

Mario: 


OBJECTS  SEEN 

POSITION 

Mario 

far  left,  faring  right 

Groundl.l 

all 

QB.l 

center,  within  reach 

QB.2 

just  right  of  QB.l,  within  reach 

QB.3 

right  of  QB.2  too  high 

QB.4 

just  right  of  QB.3,  too  high 

G.l 

far  right  (moving  left) 

Scaf2.1 

under  QB.4 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Mario  moves  right 

ambiguous 

press-right-button 

F.L.O.  CONSISTENT 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

WITH 

Comparison 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

to  Model 

search-in-block(QB.  1 ) 
search-in-block(QB  .2) 
search-in-b!ock(QB.3) 
search-in-block(QB  .4) 
attack-enemy(G.l) 
move-to-end 

search-in-block(QB.l) 

press-right-button 

Figure  8.  Example  of  the  charts  in  Appendix  HI  comparing  the  expert's  observed 
behavior  with  the  predictions  of  the  models. 


corresponding  to  the  Display .x  label  on  each  of  the  model  charts  in  Appendix  n.  This 
designation  corresponds  roughly  to  time,  where  each  display  represents  about  a  second's 

VYVJim  VJL  ovuvujr* 

In  Figure  9,  for  each  display,  a  black  dot  marks  the  inferred  function  of  the  observed 
behavior  (from  the  INFERED  FUNCTION  box  of  Figure  8).  If  no  unique  function  can  be 
inferred  from  the  observed  behavior,  no  black  dot  appears  in  that  display  column.  A  gray 
dot  marks  the  FLO  predicted  by  the  model  (from  Figures  7  &  8).  Thin  black  circles  indicate 
the  FLOs  consistent  with  the  observed  behavior  (from  Figure  8).  To  highlight  the  predicted 
sequence  of  FLOs,  the  predicted  FLOs  (gray  dots)  are  joined  by  gray  lines;  to  highlight  the 
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the  redundant  FLOs  making  it  look  like  one  elongated  circle  of  an  FLO  (Displays.8  &  .9, 
and  Displays.23  to  .27).  When  a  function  cannot  be  inferred  from  the  observed  behavior 
and  no  black  dot  appears  in  1 1:*  display  column,  the  black  line  is  passed  through  the  gray 
dot  if  the  observed  behavior  is  consistent  with  the  predicted  FLO;  if  they  are  inconsistent, 
the  black  line  does  not  pass  through  the  gray  dot  in  that  display  column,  but  goes  straight  to 
the  next  inferred  function. 

The  sense  of  this  graphic  display  is  that  when  the  black  and  gray  lines  track  each  other,  the 
model  is  making  good  predictions  of  observed  behavior.  Where  the  black  and  gray  lines  do 
not  track,  the  model  is  not  predicting  the  observed  behavior.  Where  there  is  a  haze  of  thin 
black  circles  above  and  below  the  black  dots,  the  behavior  is  consistent  with  many  possible 
functions  and  it  is  likely  that  any  model  prediction  will  track  behavior,  where  black  circles 
are  few,  the  observed  behavior  is  consistent  with  only  a  few  functions  and  the  model  must 
be  accurate  to  track  behavior. 

Figure  10  uses  the  same  graphic  conventions  as  Figure  9,  but  for  the  keystroke-level 
behavior  and  predictions.  Each  horizontal  unit,  delineated  by  dotted  lines,  corresponds  to 
the  FLO  implemented  by  the  KLOs  shown  in  that  unit  and  also  to  the  display  that  caused  the 
selection  of  that  FLO.  Therefore,  eaci>  horizontal  unit  is  labelled  at  the  top  with  its 
corresponding  Display.x  and  at  the  bottom  with  its  associated  FLO. 

DISCUSSION  OF  THE.. GOMS.  ANALYSIS 

Some  caveats 

Before  discussing  the  implications  of  this  analysis,  three  strong  cautions  must  be  presented. 
First,  this  analysis  explores  a  short  segment  of  behavior.  Although  a  qualitative  look  at  an 
additional  segment  twice  as  long  as  the  current  analysis,  representing  traversal  of  the  rest  of 
World  1  Level  1,  indicates  that  much  of  that  behavior  is  also  predictable,  the  detailed 
analyses  and  comparisons  must  be  done  before  any  conclusions  can  be  drawn  with 
confidence. 


Second,  World  1  Level  1  is  the  simplest  level  in  the  game.  Although  it  is  not  evident  in  the 
expert  videotape,  if  Mario  stops  moving,  nothing  will  happen  most  of  the  time  at  this  level 
of  play.  Thus,  the  interaction  is  primarily  user-driven,  not  system-driven.  This  feature  is 
not  common  in  many  other  types  of  video  games  and  it  does  not  even  persist  in  Super 
Mario  Bros.  3,  as  the  higher  worlds  have  many  more  enemies  actively  seeking  to  loll 
Mario. 


Last,  in  this  part  of  World  1  Level  1  the  objective  of  the  next  cycle  is  almost  always  visible 
in  the  display.  Although  a  FLO  that  searches  through  the  world  for  blocks  to  break  or  items 
to  gather  can  be  deduced  from  the  instruction  booklet,  it  was  not  invoked  in  this  segment  of 
behavior  because  the  next  block  to  break,  item  to  gather,  or  enemy  to  attack,  was  visible  in 
26  of  the  31  displays.  Other  games,  and  other  levels  even  within  this  game,  are  not  as 
perceptually  driven.  Without  constant  triggering  of  operators  by  the  elements  in  the 
display,  the  behavior  would  take  on  a  more  searching  aspect  (in  the  case  of  no  knowledge), 
or  a  more  planful  aspect  (in  ilie  case  of  additional  knowledge  of  hidden  items). 


The  nature  of  expertise  in  these  analyses 

Before  examining  the  specifics  results  of  these  analyses,  it  is  useful  to  locate  the  use  of 
expertise  in  both  the  models  and  the  expert's  behavior.  The  GOMS  models  have  no 
knowledge  of  Mario's  world;  they  do  not  know  anything  except  what  is  on  the  screen. 
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Figure  10.  Comparison  of  keystroke-level  predictions  to  observed  behavior 

(description  same  as  that  of  Figure  9.) 
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Therefore,  they  cannot  anticipate  cither  functions  or  keystrokes.  This  is  not  true  of  the 
expert.  A  post-trial  interview  with  KP,  as  well  as  observation  of  several  trials  of  play, 
clearly  showed  that  he  knows  every  detail  of  World  1  Level  1.  He  knows  where  die 
treasures  are  hidden  and  what  super  powers  they  bestow.  He  knows  when  and  where 
enemies  will  appear  and  whether  each  enemy  should  be  killed  or  whether  it  can  be  ignored. 
His  actions  in  two  situations  indicate  that  he  anticipates  functions  to  perform  with  elements 
not  yet  on  the  display.  These  situations  are  when  he  jumps  to  catch  the  mushroom  before  it 
emerges  from  QB.4  and  when  he  starts  his  take-off  to  fly  along  a  path  of  coins  leading  into 
the  sky;  before  that  path  appears  on  the  display.  It  is  not  surprising  that  an  expert  has  such 
knowledge,  only  that  this  world  requires  him  to  use  it  so  infrequently.  The  models  make 
predictions  without  reference  to  it  at  all. 

One  the  other  hand,  the  GOMS  models  are  expert  in  their  execution  of  operators.  KLOs  are 
always  assumed  to  accomplish  their  intended  movement.  For  example,  if  a  press-A-button 
KLO  is  used  to  jump  on  the  back  of  an  approaching  enemy,  it  succeeds,  never  missing  and 
allowing  the  enemy  to  kill  Mario.  This  is  clearly  not  the  case  with  a  human  novice  who  has 
never  used  the  Nintendo  hand-held  controller  before.  Informal  observation  of  novice 
players  shows  many  errors  in  execution:  jumps  are  not  timed  correctly  or  are  not  initiated  in 
the  correct  place  so  Mario  doesn't  kill  an  enemy  or  misses  a  block,  buttons  are  not  released 
in  time  so  Mario  runs  off  the  cliff,  etc.  In  addition,  the  selection  rules  in  these  models 
embody  expert  knowledge:  the  models  always  select  the  same  method  for  accomplishing  an 
FLO  the  expert  selects.  Since  KLOs  are  always  successful,  and  the  method  always  reflect 
an  expert  selection,  a  sequence  of  KLOs  in  service  of  an  FLO  is  always  successful  if  it 
completed  before  another  FLO  is  selected. 

Thus,  these  GOMS  models  represent  the  behavior  of  a  player  with  expert  motor 
movements,  expert  knowledge  of  methods  for  collecting  treasures  and  killing  enemies,  but 
no  knowledge  of  what  they  will  encounter  in  this  world.  This  description  coincidentally  fits 
a  human  player  who  has  played  Super  Mario  Bros,  and  Super  Mario  Bros.  2.,  but  never 
Super  Mario  Bros.  3.  Such  a  person  would  have  be  expert  in  most  methods  and  all 
keystrokes,  but  would  not  know  the  world  at  all.  Further  test  of  these  models  would 
benefit  from  observation  and  analysis  of  such  a  player. 

Highlights  of  these  analyses 

The  results  of  this  analysis  indicate  that  GOMS  can  capture  the  knowledge  necessary  to 
predict  the  course  of  this  behavior.  The  FLOs  in  this  segment  of  behavior  are  virtually 
dictated  by  some  simple  heuristics  derived  from  the  overriding  goals  of  the  game,  the 
operators  and  methods  described  in  the  instruction  booklet,  and  the  elements  visible  on  the 
display.  Of  the  31  FLOs  predicted,  21  can  be  unambiguous  inferred  from  the  observed 
behavior,  9  are  consistent  with  observed  behavior,  only  1  is  inconsistent  with  observed 
behavior,  and  no  behavior  indicates  any  functions  not  predicted  by  the  model.  At  the 
keystroke-level,  of  the  62  KLOs  predicted,  46  are  observed,  3  are  consistent  with  observed 
behavior,  12  arc  inconsistent  with  observed  behavior,  and  35  keystrokes  are  observed  but 
not  predicted. 

Figure  9,  the  comparison  of  the  observed  behavior  to  the  predicted  FLOs  show  some 
interesting  features  of  the  analysis.  Most  striking  is  the  excellent  agreement  between  the 
predicted  FLOs  and  the  observed  behavior.  Only  once  in  31  opportunities  is  an  FLO 
selected  that  is  inconsistent  with  observed  behavior  (Display.21).  This  instance  is  at  a  point 
in  the  game  where  the  expert  seems  to  anticipate  gathering  some  coins  that  are  not  yet 
visible  on  the  screen  and  for  which  he  must  attain  flying  speed.  He  moves  left  to  get  a 
running  start  for  take-off.  The  model  does  not  have  the  prior  knowledge  that  coins  will 
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eventually  appear  to  the  right,  and  moves  towards  the  end  of  the  level  (to  the  right)  until  it 
sees  the  coins,  and  then  gathers  them.  Thus,  the  expert  moves  to  the  left  in  anticipation  of 
flying,  while  die  model  moves  to  the  right  until  it  sees  something  to  gather.  As  soon  as  the 
model  sees  the  coins,  the  model  and  the  expert  have  the  same  knowledge  and  they  come 
back  into  synch. 

A  second  interesting  feature  is  that  in  4  of  the  1 1  displays  in  which  the  expert  behavior  does 
not  match  the  predicted  behavior  (Displays.10,  16, 18,  and  21),  the  predicted  FLO  is 
move-towaids-end.  This  is  the  default  FLO  used  when  there  are  no  elements  on  the  display 
to  trigger  the  proposal  of  other  FLOs.  In  effect,  the  model  is  exploring  the  world  when  it 
selects  this  FLO.  The  expert,  unlike  the  model,  knows  this  part  of  the  world  quite  well  and 
anticipates  the  upcoming  elements.  The  expert  does  not  display  this  exploring  behavior,  but 
anticipates  the  next  function  and  starts  to  perform  it  immediately.  Thus,  if  the  expert's 
cognitive  processes  were  more  clear  from  the  verbal  protocol,  we  expect  more  goal-directed 
behavior  to  emerge  than  is  predicted  by  the  model.  However,  if  a  novice  were  to  play  the 
game,  with  no  knowledge  of  what  comes  next  (like  the  model),  the  model  predicts  that  this 
exploring  behavior  would  be  evident 


A  third  interesting  feature  is  that  only  8  of  the  31  displays  had  observed  behaviors 
consistent  with  several  FLOs  simultaneously  (Displays  0, 7, 8, 10, 23, 24, 25,  and  26). 
This  means  that  uie  predicted  FLOs  must,  be  exactly  in  line  with  the  expert's  goals  to  make  a 
correct  prediction.  The  behavior  can  be  interpreted  only  as  belonging  to  a  single  function 
most  of  the  time,  so  picking  an  FLO  with  incorrect  preferences,  or  at  random,  would  rarely 
result  in  a  match  to  observed  behavior.  This  indicates  that  the  function-level  model  is  quite 
good  at  predicting  the  sequence  of  functions  an  expert  will  perform  in  this  phase  of  the 
game. 

The  comparison  of  keystrokes  inferred  from  the  observed  behavior  and  the  KLOs  predicted 
by  the  keystroke-level  model.  Figure  10,  also  has  several  interesting  features.  The  most 
striking  is  the  dramatic  difference  in  predictive  power  between  the  function-level  and  KLO 
levels;  the  function-level  predictions  are  almost  perfect  while  the  keystroke-level  model 
predicts  only  about  half  of  the  observed  behavior  (46  of  the  96  observed  keystrokes).  This 
is  the  same  pattern  of  results  obtained  in  Card,  Moran,  and  Newell's  analyses  of  text  editing 
(1983),  with  FLOs  predicting  close  to  100%  of  text-editing  behavior  and  KLOs  predicting 
about  60%. 

Such  a  large  drop  in  predictive  power  indicates  that  the  keystroke-level  models,  both  for  the 
text-editing  task  and  the  game-playing  task,  may  be  missing  important  features  that 
correspond  to  the  unexplained  behavior.  For  the  text-editing  task,  Card,  Moran,  and 
Newell  state  that  much  of  the  unexplained  behavior  involved  hand  movements  outside  the 
model,  e.g.  the  user  licked  her  fingers  before  turning  each  page,  an  act  not  included  in  their 
model.  In  this  analysis,  there  seems  to  be  a  similar  type  of  unexplained  behavior,  twisting 
while  jumping. 


Twenty-four  of  the  37  observed  keystrokes  not  predicted  involve  turning  Mario  left  or  right 
while  jumping  or  floating.  This  twisting  motion  may  be  in  service  of  a  goal  not  represented 
in  this  model,  e.g.  displaying  flashy  behavior  to  impress  oilier  players.  This  hypothesis 
directs  research  attention  to  the  social  aspects  of  video  game  play  to  discover  goals  not 
evident  from  the  instruction  booklet.  Alternatively  this  behavior  may  serves  to  slow 
Mario's  horizontal  motion  while  in  the  air.  The  second  hypothesis,  which  we  have  recently 
been  assured  by  other  experts  is  the  correct  explanation,  indicates  that  the  motor  model  used 
in  this  analysis  is  too  simple  to  capture  the  interaction  of  forces  (e.g.  the  equivalents  of 
gravity,  friction,  and  air  resistance)  •  in  Mario's  world.  A  more  detailed  perceptual-motor 
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model  should  include  duration  of  button  presses,  perceptual  monitoring,  and  the  playei 
prediction  of  effects. 

Questions  of  learning 

Production-system  formulations  of  GOMS  have  been  used  successfully  to  predict  learning 
of  several  domains  (e.g.,  Kieras  &  Bovair,  1986;  Ziegler,  et  al,  1986;  Lee,  et  al.t  1989; 
Singley  &  Anderson,  1989),  and  may  be  useful  for  understanding  learning  in  this  domain 
as  well.  The  learning  predicted  in  the  previous  studies  involved  reading  instruct'  ms  and 
internalizing  performance  rules  based  on  those  instructions,  plus  practice  with  error 
feedback.  That  process  describes  the  learning  necessary  to  perform  the  actions  and 
methods  described  in  the  instruction  booklet.  However,  several  other  types  of  learning  are 
evident  in  this  interaction. 

There  is  a  strong  component  of  motor  learning;  a  novice  continuously  overshoots  or 
undershoots  runs  and  jumps,  while  an  expert  manipulates  Mario  more  precisely.  GOMS  is 
not  currently  capable  of  predicting  motor  learning. 

The  other  obvious  learning  behavior  is  "leaming-by-dying",  that  is,  Mario  dies  because  of 
strategic  mistakes  and  the  player  leams  to  avoid  those  mistakes  in  the  future.  For  instance, 
the  method  of  kicking  a  Koopa  shell  into  a  block  on  the  ground  to  break  it  open  is  described 
in  the  instruction  manual.  When  a  novice  performs  this  action  on  QB.6,  the  Koopa  shell 
bounces  off  the  block,  right  into  Mario.  This  event  happens  too  quickly  for  the  immediate 
interaction  cycle  to  produce  adaptive  behavior,  and  Mario  dies.  The  novice  then  leams  to 
modify  the  method  to  include  jumping  out  of  the  way  as  soon  as  Mario  kicks  the  shell.  If 
the  conditions  surrounding  each  death  are  added  to  the  condition  of  a  production  rule,  and 
are  combined  with  the  appropriate  avoidance  tactic  as  the  action  of  that  production  rule,  new 
rales  could  be  added  to  tire  knowledge  in  the  GOMS  analysis.  Previous  work  with  Soar, 
modeling  strategy  change  in  a  developmental  task  (Newell,  1990)  and  modeling  recovery 
from  errors  (Laird,  1988)  indicates  that  the  architectural  mechanisms  are  sufficient  to  learn 
such  knowledge  when  the  learner  can  set  the  pace  of  learning.  This  situation  raises  the 
question  as  to  whether  those  same  mechanisms  will  be  able  to  learn  when  the  feedback 
occurs  within  the  immediate  interaction  cycle.  Another  question  is  whether  a  regularity 
similar  to  the  30  seconds  per  operation9  found  in  previous  studies  of  learning  and  transfer 
would  emerge  from  this  interactive  style  of  learning. 

Other  forms  of  learning  produce  behavior  in  the  service  of  elements  not  yet  visible  on  the 
display  (discussed  in  the  comparison  between  observed  functions  and  predicted  FLOs).  At 
least  two  types  of  learning  might  occur  to  produce  this  behavior.  The  expert  seems  to 
anticipate  elements  before  they  are  visible,  so  he  might  have  learned  cues  that  are  visible 
prior  to  the  elements  of  interest  Alternatively,  KP  says  "get  the ...  mushrooms  so  you  can 
turn  into  Super  Mario",  an  indication  of  a  goal  above  the  FLO  gather-item(mushroom);  such 
higher  level  goals  might  serve  to  combine  FLOs  into  methods  themselves.  These 
conjectures  raise  many  questions.  What  knowledge  structure  is  necessary  to  store  and 
access  goals  in  advance  of  directly  relevant  elements  on  the  display?  What  role  might  a 
hierarchy  of  goals  above  the  FLOs  play  in  anticipatory  behavior.  How  can  the  rapid 
interaction  observed  here,  with  a  new  FLO  occurring  every  second  and  behavior  being  so 
time-critical  and  difficult  that  a  verbal  protocol  cannot  keep  up,  allow  processing  of  the 
behavior  to  create  these  higher  level  goals,  or  store  cues  for  anf '  tipation  of  goals?  Does 
such  processing  happen  in  the  course  of  the  game,  or  is  there  only  leaming-by-dying 


9  Soar  operators  are  equivalent  to  the  productions  used  in  other  research.  Soar  productions  are  at  a  lower 
grain-size  and  would  not  be  expected  to  correspond  to  the  learning  times  found  previously. 
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(where  the  user  reflects  on  what  happened  only  when  the  immediate  interaction  cycle  is 
broken  by  Mario's  death),  or  perhaps  learning  occurs  only  even  further  outside  the  confines 
of  the  game  in  discussion  with  other  players  or  while  reading  "Nintendo  Power"10  (a 
magazine  containing  game  hints)?  How  can  Soar's  chunking  mechanism  produce  this 
anticipatory  behavior? 

An  interesting  prediction  about  what  is  and  is  not  learr,ed  arises  from  the  environmentally 
driven  characteristic  of  this  portion  of  the  game.  Since  expert  knowledge  about  upcoming 
elements  in  the  world  is  not  vsed  in  the  selection  of  FL0s„  different  experts  would  follow 
the  same  FLO  sequence,  and  even  novices  would  produce  the  same  function-level  behavior. 
The  motor  operations  would  differ,  especially  for  novices  who  are  not  able  to  manipulate 
Mario  reliably,  but  the  functions  they  tiy  to  accomplish  (e.g.  search-in-block(QB.l)  then 
search-in-block(QB.2))  would  not  be  a  learned  sequence,  rather  they  would  arise  at  all 
levels  of  skill  from  the  elements  on  the  display.  Although  verification  of  this  prediction 
remains  an  open  issue,  informal  observation  of  an  additional  expert  and  several  novices  do 
not  contradict  this  prediction. 

Speculation  about  motivation 

Perhaps  the  most  striking  feature  of  video  games  is  that  people  play  them  for  hours  on  end. 
Empirical  studies  of  computer  games  have  produced  speculation  about  the  factors 
contributing  to  motivation  (Malone,  1980),  but  as  yet  GOMS  has  not  been  used  to  explore 
this  aspect  of  user-computer  interaction.  Based  on  the  form  and  process  of  this  GOMS 
analysis,  we  present  some  purely  speculative  words  about  why  this  game  is  so  absorbing. 
First,  the  perceptually-based  production  system  form  of  this  model  allows  for  the  notion  of 
capture.  That  is,  since  cues  in  the  environment  trigger  operator  proposal  and  selection, 
whenever  those  cues  are  present,  those  operators  will  be  selected  and  executed.  Thus,  the 
environment  captures  the  player  and  sends  him  or  her  looping  through  the  immediate 
interaction  cycle.  Second,  the  rate  of  goal  satisfaction  (i.e.,  FLO  accomplishment)  may  be 
related  to  this  game's  ability  to  absorb  its  players.  If  the  rate  is  too  high,  the  game  is  too 
easy  and  boring  (e.g.,  KP  yearned  to  go  on  to  a  higher  level);  if  the  rate  is  too  low,  the 
game  becomes  frustrating;  if  it  is  just  right,  the  game  captures  players  for  hours  on  end. 
Thus,  other  open  research  questions  are  whether  these  concepts  of  capture  and  rate  of  goal 
satisfaction  have  any  relation  to  what  we  know  as  fascination,  or  motivation,  and,  if  so, 
whether  the  parameters  of  these  concepts  can  be  discovered  and  eventually  manipulated  to 
predict  the  fascination  level  of  new  games,  educational  programs,  and  other  application 
interfaces. 

Directions  for  future  research 

The  stated  purpose  of  the  original  panel,  and  this  report,  is  to  demonstrate  GOMS  analyses 
in  the  area  of  highly  interactive  tasks  and  encourage  researchers  and  practitioners  to  use 
GOMS  in  their  work.  We  see  three  directions  in  which  future  work  may  spring  from  our 
analyses. 

The  first  direction  is  the  extension  of  this  particular  analysis.  The  panel  presentations 
represented  a  first  pass  GOMS  analysis  for  each  of  the  participants.  This  paper  represents  a 
"pass  and  a  half'  for  our  analyses,  i.e.,  cleaned  up  but  not  substantially  changed  from  the 
first  pass.  A  second  pass  would  tackle  issues  raised  by  the  panel  participants:  perceptual 
monitoring  of  the  effects  of  motor  operations,  a  more  detailed  model  of  motor  operators  and 
their  interactions  with  the  forces  designed  into  Mario's  world,  examination  of  the  operator 


10  Nintendo  Power  is  a  trademark  of,  and  is  under  copyright  to,  Nintendo  of  America. 
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durations  and  comparison  to  the  human  time  course  of  behavior.  A  second  pass  would  also 
include  consideration  of  other  research  in  the  domain,  notably  the  AI  programs  that  play  the 
video  games  Pengo  and  Amazon  (Pengi  in  Agre  &  Chapman,  1987,  and  Sonja  in 
Chapman,  1990,  respectively).  Although  these  programs  were  never  intended  to  be 
cognitive  models  of  human  behavior,  and  were  neve  *  formally  compared  to  human 
behavior,  they  provide  insight  into  a  simple  architecture  that  uses  relatively  little  game 
knowledge  but  is  sufficient  for  performing  the  immediate  interaction  necessary  to  play  the 
game. 

A  second  direction  is  for  HF  practitioners  to  use  GOMS  to  make  predictions  of  human 
behavior  with  the  systems  they  help  to  design.  At  first,  these  analyses  may  be  done  in 
parallel  with  other  HF  methods,  e.g.  user  trials,  to  validate  the  analytic  predictions  (see 
Gray,  et.  al.  1990  for  an  example).  In  time,  we  expect  that  confidence  in  GOMS  analyses 
will  grow  and  allow  design  decisions  to  be  made  in  advance  of  the  working  prototypes  (or 
Wizard-of-Oz  mock-ups)  and  extensive  user  testing. 

The  third  direction  is  to  tackle  the  limitations  of  GOMS  and  extend  the  analysis  technique 
itself.  As  presented  above,  GOMS  is  an  evolving  analytic  technique.  Its  original 
formulation  presented  by  Card,  Moran  and  Newell  (1980, 1983)  has  undergone  several 
changes  to  extend  the  original  scope,  most  recently  this  includes  a  change  in  the  underlying 
architecture  to  the  Soar  unified  theory  of  cognition.  The  change  to  Soar  has  allowed  GOMS 
to  encompass  the  interruption  of  tasks  and  responsiveness  to  the  environment  (as  begun  in 
this  analysis  and  in  John,  et.  al.,  1990).  We  expect  this  change  will  lead  to  increased 
understanding  and  prediction  of  learning,  errors,  knowledge  representations,  and  other 
issues  of  interest  to  the  HF  and  HCI  community,  as  well  as  to  the  cognitive  science  and 
cognitive  psychology  communities. 

CONCLUSIONS 

This  analysis  demonstrates  GOMS’s  power  to  predict  behavior  in  a  highly  interactive 
domain.  It  shows  that  GOMS  is  especially  useful  for  making  predictions  of  human 
behavior  at  the  function  level.  It  demonstrates  how  even  an  initial  GOMs  analysis  can  help 
direct  research  effort  by  focusing  on  areas  of  the  model  that  do  not  predict  human 
performance  well.  Further,  it  can  make  predictions  about  the  general  character  of  a  task, 
e.g.,  that  novices  and  experts  alike  will  have  the  same  function-level  goal  structure  in  this 
environmentally  driven  interaction. 

This  preliminary  investigation  also  poses  many  questions.  In  pursuit  of  answers  to  those 
questions,  this  domain  seems  rich  in  behavioral  phenomena,  affording  opportunities  to 
explore  the  immediate  interaction  cycle  with  its  rapid  perception,  cognition  and  motor 
actions,  several  aspects  of  learning,  and  perhaps  even  the  elusive  concept  of  fascination.  In 
addition,  research  into  the  validation  of  GOMS  analysis  in  real-world  settings  and  the 
extension  of  the  GOMS  analytic  technique  would  be  valuable  contributions  to  HF  and  HCI 
methodologies. 
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Appendix  I  -  Transcription  of  Observed  Behavior 
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Transcript  supplied  by  Wayne  Gray  Inferences  made  during  our  analysis 

TIME  (SEC.)  INFERRED  CORRESPONDING 
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Display.  0  ( Start-up  screen >  Mario  starts  at  far  left  and  Goomba  is  at  far  right  moving  left. 

Mario  is  trying  to  get  the  first  question  block. 


Display.  1  Goomba  gets  threateningly  close  to  Mario,  so  Mario  attacks  it. 


Display.  2  Mario  goes  on  10  get  the  closest  question  block  (QB.l). 
State  of  Mario:  Small  Mario 


_ _ _ ACHIEVED _  _ 

Application  METHOD  NECESSARY  MOVES  BY  K.L.O.  K.L.O.  PERFORMED 

of  Selected  hit-fiom-bottom  move-horizontal  (until  under)  - >  press-right-button  performed 

F.L.O. _ iump _  _ >  press-A-button _  I  performed _ _J  < — Triggers  Next  Display 


Display.  4  Mario  goes  on  to  get  QB.3  but  it  is  too  high  so  he  jumps  onto  a  scafold  to  teach  it. 


ACHIEVED 

Application  METHOD  NECESSARY  MOVES  BY  K.L.O.  P  K.L.O.  PERFORMED 

of  Selected  hit-from-bottom  move-horizontal  (until  under)  - >  press-A-button  performed 

F.L.O.  jump  (onto  Scaf2.1)  - >  press-right-button  performed 

_ I iumo  fto  OB.41  _ _ >.  1 nress- A-button _  [performed _ ; _  < — Triggers  Next  Display 


Display.  10  Neither  Goomba  nor  Koopa  are  an  immediate  threat 

Mario  moves  forward  by  jumping  onto  the  scafolds. 


Display.  11  Mario  sees  block  on  the  ground  which  can  only  be  opened  by  thowing  a  shell  at  it. 

So  Mario  stomps  on  the  Koopa  Trooper  in  order  to  get  a  shell  to  throw  at  the  block. 
Then  Mario  throws  the  shell  at  QB.6  and  breaks  it 


Display.  12  Mario  goes  to  get  the  leaf  that  popped  out  of  the  question  block  (QB.6)  so  that  he  can  turn  into  Racoon  Mario. 

State  of  Mario:  Super  Mario 


Display.  14  There  are  no  items  or  enemies  on  the  screen  so  Mario  just  moves  toward  the  end  of  the  level. 


Display.  16  Again,  there  are  no  items  or  enemies  on  the  screen  so  Mario  just  moves  toward  the  end  of  the  level. 


Display.  18  Again,  then*,  are  no  items  or  enemies  on  the  screen  so  Mario  just  moves  toward  the  end  of  the  level. 


Display.  20  The  para-goomba  turns  into  a  goomba  (G 5)  after  Mario  stomps  on  it,  so  he  has  to  attack  it  again. 

State  of  Mario:  Racoon  Mario 


Display.  24  A  second  coin  comes  into  view;  this  one  is  over  the  cliff  and  too  high  for  Mario  to  jump. 


Display.  26  Mario  arrives  under  Coin.  1  and  jumps  to  get  it 


Display.  27  Mario  cannoi:  get  the  second  coin  by  jumping  to  it  so  he  has  to  back-up  and  fly  to  it. 


Display..  38  After  getting  the  second  coin,  Mario  continues  flying  to  the  third  coin,  and  a  fourth  and  fifth  coin  come  into  view  which  he  can  also  fly  to. 


: —  Triggers  Next  Display 


Appendix  m  -  Comparison  between  Observed  Expert  Behavior 
and  Predictions  of  the  GOMS  Models 


Display.  0  ( Start-up  screen) 


Mario  starts  at  far  left. 

Goomba  is  at  far  right  moving  left 
Mario  is  trying  to  get  the  first  question  block. 


State  of  Mario:  Small  Mario 


OBJECTS  SEEN 

POSITION 

Mario 

far  left,  facing  right 

Groundl.l 

all 

QB.l 

center,  within  reach 

QB.2 

just  right  of  QB.l,  within  reach 

QB.3 

right  of  QB.2  too  high 

QB.4 

just  right  of  QB.3,  too  high 

G.l 

far  right  (moving  left) 

Scaf2.1 

under  QB.4 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Mario  moves  right 

ambiguous 

press-right-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

Comparison 
to  Model 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

search-in-block(QB.l) 

search-in-bloclc(QB.2) 

scaich-in-block(QB.3) 

scarch-in-block(QB  .4) 

attack-cnemy(G.l) 

move-to-end 

search-in-block(QB  .1) 

press-right-button 

ffl-2 


Display.  1  Goomba  gets  threateningly  close  to  Mario,  so  Mario  attacks  it 

State  of  Mario:  Small  Mario 


OBJECTS  SEEN 

POSITION 

Mario 

center,  facing  right 

Groundl.l 

all 

G.l 

moving  left  close  to  Mario 

QB.l 

center,  within  reach 

QB.2 

just  right  of  QB.l,  within  reach 

QB.3 

right  of  QB.2,  too  high 

QB.4 

just  right  of  QB.3,  too  high 

Scaf2.1 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Mario  jumps  to  the  right 

press-right-button 

press-A-button 

turns  left  at  apogee 

press-left-button 

lands  on  G.l 

attack-enemy(G.l) 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KX.O.  PREDICTED 

Comparison 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

to  Model 

attack-enemy(G.l) 

attack-enemy(G.l) 

press-A-button 

Display.  2 
State  of  Mario: 


On 

Screen 


Expert's 

Behavior 


Comparison 
to  Model 


m-3 


Mario  goes  on  to  get  the  closest  question  block  (QB.l). 
Small  Mario 


OBJECTS  SEEN 

POSITION 

Mario 

center,  facing  left 

Ground  1.1 

all 

QB.l 

center,  within  reach 

QB.2 

just  right  of  QB.l,  within  reach 

QB.3 

right  of  QB.2  too  high 

QB.4 

just  right  of  QB.3,  too  high 

Scaf2.1 

under  QB.4 

Scaf3.2 

just  to  the  right  of  Scaf2.1 

OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Mario  jumps  off  G.l 

press-A-button 

turns  to  the  right  at  apogee 

press-right-button 

jumps  again 

press-A-button 

turns  to  left  and  hits  QB.l 

scarch-in-block(QB.l) 

press-left-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

search-  in-block(QB .  1 ) 

search-in-block(QB.l) 

press-right-button 

press-A-button 

in-4 


Display.  3  Mario  goes  on  to  get  the  next  question  block  (QB.2) 

State  of  Mario:  Small  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

center,  facing  left 

Screen 

Groundl.l 

aU 

QB.2 

just  right  of  QB.l,  within  reach 

QB.3 

right  of  QB.2  too  high 

QB.4 

just  right  of  QB.3,  too  high 

Scaf2.1 

under  QB.4 

ScaO.2 

just  to  the  right  of  Sca£2.1 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Mario  turns  to  right  on  ground 

press-right-button 

jumps 

press-A-button 

turns  to  left  and  hits  QB.2 

search-in-block(QB  .2) 

press-left-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJLO.  PREDICTED 

Comparison 
to  Model 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

search- in-block(QB  .2) 

search-in-block(QB  2) 

press-right-button 

press-A-button 

m-5 


Display.  4  Mario  goes  on  to  get  QB.3  but  it  is  too  high  so  he  jumps  onto  a  scafold  to  reach  it 
State  of  Mario:  Small  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

center,  facing  left 

Screen 

Groundl.l 

all 

QB.3 

right  of  QB.2  too  high 

QB.4 

just  right  of  QB.3,  too  high 

Scaf2.1 

under  QB.4 

Scaf3.2 

just  to  the  right  of  Scaf2.1 

Expert’s 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Mario  turns  to  right  on  ground 

press-right-button 

jumps 

prcss-A-button 

turns  left  and  lands  on  Scaf2.1 

press-left-button 

jumps 

press-A-button 

turns  to  right  and  hits  QB.3 

search-in-block(QB.3) 

press-right-button 

Comparison 
to  Model 


F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

scarch-in-block(QB.3) 

search-in-block(QB  .3) 

press-right-button 

press-A-button 

press-left-button 

press-A-button 

m-6 


Display.  5  From  the  scafold  he  can  also  reach  QB.4. 

State  of  Mario:  Small  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

on  ground  under  QB.3,  facing  right 

Screen 

Ground  1.1 

all 

QB.4 

just  right  of  QB.3,  too  high 

Scaf2.1 

under  QB.4 

Scaf3.2 

just  to  the  right  of  Scaf2J 

Plant  1 

far  right 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

turns  to  left  and  lands  on  ground 

prcss-left-button 

turns  to  right 

press-right-button 

jumps 

press-A-button 

turns  left  in  air 

prcss-left-button 

turns  right  in  air 

press-right-button 

turns  left  and  lands  on  Scaf2.1 

press-left-button 

jumps 

press-A-button 

turns  to  right  and  hits  QB.4 

search-in-block(QB  .4) 

press-right-button 

Comparison 
to  Model 


F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

search-in-block(QB  .4) 

search-in-block(QB  .4) 

press-right-button 

press-A-button  (to  Scaf2.1) 

press-A-button  (hit  QB.4) 

m-7 


Display.  6  Mario  jumps  up  to  get  the  mushroom  which  pops  out  from  QB.6 

before  it  even  starts  emerging  from  the  question  block. 

State  of  Mario:  Small  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

on  Scaf2.1,  facing  right 

Screen 

Ground  1.1 

all 

Mush.l 

above  QB.4 

Scaf2.1 

under  QB. 4 

ScaO.2 

just  to  the  right  of  Scaf2.1 

Plant  1 

far  right 

OBSERVED 

INFERRED 

INFERRED 

Expert's 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Behavior 

turns  left  and  lands  on  Scaf2.1 

prcss-left-button 

jumps  to  Mush.l 

gather-item(Mush.l) 

press-A-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

Comparison 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

to  Model 

gather-item(Mush.l) 

gather-item(Mush.  1 ) 

press-right-button 

(run  into  Mush.l) 

m-8 


Display.  7  Mario  jumps  from  QB.4  onto  Scaf3.2,  avoiding  the  fireball. 

State  of  Mario:  Small  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

on  QB.4,  facing  left 

Screen 

Ground  1.1 

all 

Scaf2.1 

under  QB.4 

ScaD.2 

just  to  the  right  of  Sca£2.1 

Plant  1 

right 

Fireball 

coming  toward  Mario 

Scaf2.3 

to  the  right  of  Plant  1 

OBSERVED 

INFERRED 

INFERRED 

Expert's 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Behavior 

moves  right  landing  on  Scaf3.2 

avoid-dan  ger(Fireball) 

press-right-button 

Comparison 
to  Model 


F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

avoid-danger(Fireball) 

avoid-danger(Fireball) 

press-right-button 

move-toward-end 

ffl-9 


Display.  8  Mario  goes  after  the  next  question  block  by  jumping  over  the  plant  and  onto  Scaf2.3. 


State  of  Mario:  Super  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

on  Scaf3.2,  facing  right 

Screen 

Groundl.l 

all 

Plant  1 

right 

Scaf2.3 

right  of  Plant  1 

QB.5 

above  Scaf2.3 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Mario  jumps  over  plant 

press-A-button 

turns  left  and  lands  on  3caf2.3 

ambiguous 

prcss-left-button 

Comparison 
to  Model 


F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KX.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BV  MODEL 

BY  MODEL 

search-in-block(QB  .5) 

search-in-block(QB-5) 

press-A-button 

avoid-danger(PlanLl) 

press-right-button 

move-toward-cnd 

press-A-button 

press-right-button 

press-right-button 

m-io 


Display.  9  Goomba  appears  before  Mario  has  reached  QB.5. 

It  is  not  an  immediate  threat  so  Mario  continues  toward  QB.5. 

State  of  Mario:  Super  Mario 


Ob 

Screen 

OBJECTS  SEEN 

Mario 

Ground  1.1 

Plant.  1 

Scaf2.3 

QB.5 

G.5 

POSITION 

on  Scaf2.3  under  QB.5,  facing  left 
all 

far  left 

right  of  Plant.1 
above  Scaf2.3 

on  Ground  1.1  in  front  of  Scaf2.3 

OBSERVED 

INFERRED 

INFERRED 

Expert's 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Behavior 

turns  to  the  right 
jumps  to  QB.5 

search-in-block(QB  5) 

press-right-button 

press-A-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

Comparison 
to  Model 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

search-in-block(QB  5) 

search-in-block(QB.5) 

press-right-button 

press-A-button 

m-ii 


Display.  10  Neither  Goomba  nor  Koopa  are  an  immediate  threat  and  Mario  drops  down  onto  Scaf  1 .6. 
State  of  Mario: 


On 

Screen 


Expert's 

Behavior 


Comparison 
to  Model 


F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

attack-cnemy(K.l) 

move-toward-end 

press-right-button 

move-toward-end 

press-A-button 

press-right-button 

press-A-button 

OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

jumps 

press-A-button 

turns  left 

press-left-button 

turns  rigid  and  lands  on  Scaf  1.6 

ambiguous 

press-right-button 

Super  Mano 


OBJECTS  SEEN 

POSITT  N 

Mario 

on  Scaf2.3,  facing  right 

Groundl.l 

all 

Scaf2.3 

far  left 

Scaf3.4 

center 

Scaf4i 

right 

Scaf l.o 

below  Scaf 4 .5 

G.5 

on  Groundl.l  in  front  of  scafolds 

K.1 

on  Scafl.6 

ni-12 


Display.  11  Mario  sees  QB.6  on  the  ground  which  can  only  be  opened  by  thowing  a  shell  at  it 
So  Mario  stomps  on  the  Koopa  Trooper  in  order  to  get  a  shell  to  throw  at  the  block. 
Then  Mario  throws  the  shell  at  QB.6  and  breaks  it 


State  of  Mario:  Super  Mario 


Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

jumps  straight  up 

prcss-A-button 

moves  right  in  air 

press-right-button 

turns  left  and  lands  on  K.l 

press-left-button 

jumps,  landing  left  of  the  shell 

press-A-button 

turns  right 

press-right-button 

picks  up  Koopa  shell 

hold-B-button 

kicks  Shell  at  QB.6 

release-B-button 

jumps  out  of  way  of  Koopa  shell 

search-in-block(QB  .6) 

rclcase-A-button 

Comparison 
to  Model 


Note:  The  last  press-A-button  is  to  jump  out  of  the  way  of  the  richocheHng  Koopa  shell 

This  KLO  is  added  to  the  hit-with-shell  method  based  on  prior  experience  with  being  killed 
by  the  bouncing  Koopa  shell  See  the  text  for  more  explanation  of  this  learning  process. 


ni-i3 


Display.  12  A  leaf  pops  out  of  the  QB.6. 

Mario  goes  to  get  it  so  he  can  turn  into  Racoon  Mario. 


State  of  Mario: 

Super  Mario 

OBJECTS  SEEN 

POSITION 

On 

Mario 

in  air,  facing  right 

Screen 

Ground  LI 

left 

Scaf3.4 

far  left 

Scaf4.5 

center 

Scafl.6 

below  Scaf43 

G.5 

on  Groundl.l  in  front  of  scafolds 

Ground2.2 

right 

QB.6  (empty) 

right,  on  Ground2.2 

Leaf.l 

over  QB.6 

OBSERVED 

INFERRED 

INFERRED 

Expert's 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Behavior 

Mario  moves  to  die  right 

gather-item(Leaf.l) 

press-right-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KX.O.  PREDICTED 

Comparison 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

to  Model 

gather-iiem(Leaf.l) 

gather-item(Leaf.l) 

press-right-button 

m-14 


Display.  13 
State  of  Mario: 


On 

Screen 


Expert's 

Behavior 


Comparison 
to  Model 


Mario  goes  to  get  the  next  question  block  (QB.7) 
Racoon  Mario 


OBJECTS  SEEN 

POSITION 

Mario 

between  QB.6  and  QB.7,  facing  right 

Groundl.l 

left 

Scafl.6 

far  left 

Ground2.2 

right 

QB.6  (empty) 

center,  on  Ground2.2 

QB.7 

in  air  to  the  right  of  QB.6 

OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

moves  right  to  QB.7 

press-right-button 

jumps 

press-A-button 

turns  left  and  hits  QB.7 

search-in-block(QB  .7) 

press-left-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KX.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

search-in-block(QB  .7) 

search-in-block(QB.7) 

press-right-button 

press-A-button 

m-i5 


Display.  14  There  are  no  items  or  enemies  on  the  screen  so  Mario  just  moves  toward  the  end  of  the  level. 
State  of  Mario:  Racoon  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

to  the  right  of  QB.7,  facing  left 

Screen 

Ground2.2 

all 

QB.6  (empty) 

far  left 

QB.7  (empty) 

left 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

moves  right 

move- to  ward-end 

press-right-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

Comparison 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

to  Model 

move-toward-end 

move-toward-end 

press-right-button 

ffl-16 


Display.  IS 

State  of  Mario: 


On 

Screen 


Expert's 

Behavior 


Comparison 
to  Model 


A  goomba  (G.3)  moves  toward  Mario  from  the  right  (becoming  an  immediate  threat), 
so  Mario  attacks  him. 

Racoon  Mario 


OBJECTS  SEEN 

POSITION 

Mario 

to  the  right  of  QB.7,  facing  right 

Ground2.2 

aH 

QB.7  (empty) 

far  left 

G.3 

moving  toward  Mario  from  the  right 

OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

turns  left 

press-left-button 

turns  right 

press-right-button 

jumps 

prcss-A-button 

turns  left  and  lands  on  G.3 

attack-enemy(G.3) 

press-left-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

attack-enemy(G.3) 

attack-encmy(G.3) 

press-A-button 

m-i7 


Display.  16  This  is  not  a  distinct  display  in  the  expert's  protocol. 

Since  the  expert  has  moved  away  from  QB.7  instead  of  waiting  for  G.3  to  approach 
(as  the  model  does),  G.4  appears  on  the  screen  before  the  expert  has  stomped  G.3. 


Display.  17  Another  goomba  (G.4)  moves  toward  Mario  from  the  right  (becoming  an  immediate  threat), 
so  Mario  attacks  him. 


State  of  Mario: 

Racoon  Mario 

OBJECTS  SEEN 

POSITION 

On 

Mario 

to  the  right  of  QB.7,  facing  left 

Screen 

Ground2.2 

all 

G.4 

moving  toward  Mario  from  the  right 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

turns  right 

press-right-button 

jumps 

press-A-button 

turns  left  at  apogee 

press-left-button 

turns  right  and  lands  on  G.4 

attack-enemy(G.4) 

press-right-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KX.O.  PREDICTED 

Comparison 

OBSERVED  BEHAVIOR 

BV  MODEL 

BY  MODEL 

to  Model 

attack-enemy(G.4) 

attack-cnemy(G.4) 

press-A-button 

m-i8 


Display.  18  This  is  not  a  distinct  display  in  the  expert's  protocol. 

Since  the  expert  has  continued  to  move  right  instead  of  waiting  for  enemies  to  approach 
(as  the  model  does),  PG.l  appears  on  the  screen  before  the  expert  has  stomped  G.4. 


Display.  19  A  para-goomba  (PG.l)  moves  toward  Mario  from  the  right  (becoming  an  immediate  threat), 
so  Mario  attacks  him. 


State  of  Mario: 

Racoon  Mario 

OBJECTS  SEEN 

POSITION 

On 

Mario 

to  the  right  of  QB.7,  facing  right 

Screen 

Ground2.2 

all 

PG.l 

moving  toward  Mario  from  the  right 

Expert’s 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

turns  left 

press-lcft-button 

jumps  (higher  than  flying  PG.l) 

press- A-button  (repeatedly) 

turns  right 

press-right-button 

turns  left  and  lands  on  PG.l 

attack-enemy  (PG.  1 ) 

press-left-button 

Comparison 
to  Model 


F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KX.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

attack-enemy  (PG .  1 ) 

attack-enemy  (PG.  1 ) 

press-A-button 

m-i9 


Display.  20  The  paragoomba  turns  into  a  goomba  (G.5)  after  Mario  stomps  on  it,  so  he  attacks  it  again. 


State  of  Mario: 

Racoon  Mario 

OBJECTS  SEEN 

POSITION 

On 

Mario 

to  the  right  of  QB.7,  facing  left 

Screen 

Ground2.2 

all 

G.5 

on  Mario's  left 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

jumps 

press-A-button 

turns  right  at  apogee 

press-right-button 

turns  left 

press-left-button 

turns  right  and  lands  on  G.5 

attack-enemy(GJ) 

press-right-button 

Comparison 
to  Model 


F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

auack-encmy(GA) 

attack-encmy(G.5) 

press-A-button 

press-lcft-button 

m-20 


Display.  21  Mario  runs  back  to  QB.6(empty)  and  then  runs  to  the  right  accelerating  to  fly. 


State  of  Mario: 

Racoon  Mario 

OBJECTS  SEEN 

POSITION 

On 

Mario 

in  center  on  Ground2.2,  facing  right 

Screen 

Ground2.2 

all 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

turns  left 

press-left-button 

jumps 

press -A-button 

lands  and  runs  to  left 

press-left-button 

turns  right 

press-right-button 

runs  to  right 

press-right-button 

accelerating 

ambiguous 

hold-B -button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KX.O.  PREDICTED 

Comparison 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

to  Model 

move-toward-end 

move-toward-end 

press-right-button 

Note:  The  first  three  observed  behaviors  ( where  Mario  turns  left,  jumps, 

and  then  runs  to  the  left)  are  not  consistent  with  any  proposed  FJ~0.s. 
The  other  three  behaviors  (where  Mario  turns  right  and  then  runs  to 
the  right  while  accelerating)  are  consistent  with  move-toward-end. 

The  whole  sequence  of  behavior  is  both  ambiguous  and  consistent 
with  move-toward-end. 


m-2i- 


Display.  22  The  first  coin  comes  into  view  as  Mario  is  picking  up  speed  to  fly. 


State  of  Mario: 

Racoon  Mario 

OBJECTS  SEEN 

POSITION 

On 

Mario 

in  center  on  Ground2.2,  facing  right 

Screen 

Ground2.2 

all 

Coin.l 

next  to  Cliff.l 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

runs  to  right 

press-right-button 

accelerating 

ambiguous 

hold-B -button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

Comparison 
to  Model 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

gather-item  (Coin.l) 

move-toward-end 

gather-item  (Coin.l) 

press-right-button 

Display.  23 


A  cliff  apears  at  the  far  right  as  Mario  moves  toward  the  coin. 


State  of  Mario:  Racoon  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

in  center  on  Ground2.2,  facing  right 

Screen 

Ground2.2 

all 

Coin.1 

next  to  Cliff.l 

Cliff.l 

far  right 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

runs  to  right 

press-right-button 

accelerating 

ambiguous 

hold-B-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

Comparison 

to  Model 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

• 

gather-item  (Coin.l) 
move-toward-end 

gather-item  (Coin.l) 

press-right-button  < 
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Display.  24  A  second  coin  con.os  into  view;  this  one  is  over  the  cliff  and  too  high  for  Mario  to  jump. 
State  of  Mario:  Racoon  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

in  center  on  Ground2.2,  facing  right 

Screen 

Ground2.2 

all 

Coin.l 

next  to  Cliff.l 

Cliff.l 

right 

Coin.2 

far  right  and  above  Coin.l 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHaVIOR 

FUNCTION 

FINGER  ACTION 

runs  to  right 

press-right-button 

accelerating 

ambiguous 

hold-B -button 

Comparison 
to  Model 


F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

gather-item  (Coin.l) 

gather-item  (Coin.l) 

press-right-butten 

gather-item  (Coin.2) 

move-toward-end 

m-24 


Display.  25  A  third  coin  comes  into  view;  this  one  is  also  over  the  cliff  and  too  high  for  Mario  to  jump. 
State  of  Mario:  Racoon  Mario 


OBJECTS  SEEN 

POSITION 

,  On 

Mario 

in  center  on  Ground2.2,  facing  right 

Screen 

Ground2.2 

all 

Coin.1 

next  to  Cliff.l 

«» 

Cliff.l 

right 

Coin.2 

right  and  above  Coin.l 

Coin.3 

far  right  and  above  Coin.2 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

runs  to  right 

press-right-button 

accelerating 

ambiguous 

hold-B -button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

Comparison 
to  Model 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

gather-item  (Coin.l) 
gather-item  (Coin.2) 
gather-item  (Coin.3) 

move-toward-end 

gather-item  (Coin.l) 

press-right-button 

O 
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Display.  26  Mario  begins  flying  up  to  Coin.  1 . 
State  of  Mario:  Racoon  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

in  center  on  Ground?  .2,  facing  right 

Screen 

Ground2.2 

all 

Coin.1 

next  to  Cliff.l 

Cliff.l 

right 

Co  in  .2 

over  Cliff.l 

Coin.3 

to  the  right  and  above  Coin.2 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

has  enough  speed  to  fly 

release-B-button 

jumps  and  flies 

press-A-button  (repeatedly) 

turns  left  and  gets  first  coin 

gather-item  (Coin.1) 

press-left-button 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

Comparison 

to  Model 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

gather-item  (Coin.1) 

gather-item  (Coin.1) 

press-right-button 

press-A-button 
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Display.  27 
State  of  Mario: 


On 

Screen 


Expert's 

Behavior 


Comparison 
to  Model 


Mario  continues  flying  to  the  second  coin. 
Racoon  Mario 


OBJECTS  SEEN 

POSITION 

Mario 

in  center  on  Ground2.2,  facing  left 

Ground2.2 

all 

Cliff.l 

right 

Coul2 

over  Cliff.l 

Coin.3 

to  the  right  and  above  Coin.2 

OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

turns  right  and  flies  to  next  coin 

gather-item  (Coin.2) 

press-right-button 
press-A-button  (repeatedly) 

F.L.O,  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

gather-item  (Coin.2) 

gather-item  (Coin.2) 

press-lcft-button 
press-right-button 
press-right-button 
hold-B -button 

relcase-B-button 

press-A-button  (repeatedly) 
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Display.  28  After  getting  the  second  coins  Mariu  continues  flying  to  the  third  co«* . 

Two  more  coins  appear  ah  c.  and  to  the  right  of  the  third  one. 

State  of  Mario:  Racoon  Mario 


OBJECTS  SEEN 

POSITION 

On 

Mario 

flying,  facing  right 

Screen 

Coin.3 

inuieair 

Coin.4 

to  the  right  and  above  Coin.3 

Coin  .5 

to  the  right  and  above  Coin.4 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

TTWCTION 

FINGER  ACTION 

flies  to  next  coin 

gather-item  (Coin.3) 

press-right-button 

press-A-button  (repeatedly) 

Comparison 
to  Model 


F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

KJL»0.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

g/her-item  (Coin.3) 

gather-item  (Coin.3) 

press-right-button 
press-A-button  (repeatedly; 

m-28 


Display.  29  After  getting  the  third  coin,  Mario  continues  flying  to  the  fourth  coin. 


State  of  Mario: 

Racoon  Mario 

OBJECTS  SEEN 

POSITION 

On 

0 

Mario 

flying,  facing  right 

Screen 

Coin.4 

in  the  air 

Coin.5 

to  the  right  and  above  Coin.4 

OBSERVED 

INFERRED 

INFERRED 

Expert's 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

Behavior 

flies  to  next  coin 

gather-item  (Coin.4) 

press-right-button 

press-  A-button  (repeatedly) 

F.L.O.  CONSISTENT  WITH 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

Comparison 

to  Model 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

-'ather-itcm  (Coin.4) 

gather-item  (Coin.4) 

press-right-button 
press-A-button  (repeatedly) 

1 
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Display.  30  After  getting  the  fourth  coin,  Mario  continues  flying  to  the  fifth  coin. 


State  of  Mario: 

Racoon  Mario 

OBJECTS  SEEN 

POSITION 

On 

Mario 

flying,  facing  right 

Screen 

Co  in  .5 

to  the  right  and  above  Coin.4 

Expert's 

Behavior 


OBSERVED 

INFERRED 

INFERRED 

BEHAVIOR 

FUNCTION 

FINGER  ACTION 

flies  to  next  coin 

gather-item  (Coin.5) 

press-right-button 

press-A-button  (repeatedly) 

Comparison 
to  Model 


F.L.~  CONSISTENT  WITH 

F.L.O.  PREDICTED 

K.L.O.  PREDICTED 

OBSERVED  BEHAVIOR 

BY  MODEL 

BY  MODEL 

gather-item  (Coin.5) 

gather-item  (Coin.5) 

press-right-button 
press-A-button  (repeatedly) 

Distribution  List 


Ms.  Lisa  B.  Achille 
Code  5530 
Naval  Research  Lab 
Overlook  Drive 
Washington,  DC  20375-5000 

Dr.  Edith  Ackermann 
Media  Labontory 
E 15.311 

20  Ames  Street 
Cambridge.  MA  02139 

Dr.  Beth  Adeieon 
Department  of  Computer  Science 
Tufts  University 
Medford,  MA  02155 

Technical  Document  Center 
AFHRL/LRr  -rDC 
Wright-Patteraon  AFB 
OH  454338503 

Dr.  Robert  Ahlera 
Code  N711 

Human  Factors  Labontoty 
Naval  Training  Systems  Center 
Orlando,  FL  32813 

Dr.  Robert  M.  Aiken 
Computer  Science  Department 
038-21 

Temple  University 
Philadelphia,  PA  19122 

Mr.  Tcjwansh  S.  Anand 
Philips  Laboratories 
345  Scarborough  Road 
BriarclifI  Manor 
New  York,  NY  10520 

Dr.  James  Anderson 
Brown  University 
Department  of  Psychology 
Providence.  R1  02912 

Dr.  John  R.  An  ’erson 
Department  of  Prycboiogy 
Carnegie-Mellon  University 
Sdienley  Park 
Pittsburgh,  PA  152U 

Dr.  Thomas  H.  Anderson 
Center  for  the  Study  of  Reading 
174  Children's  Research  Center 
51  Getty  Drive 
Champaign,  1L  (1820 

Dr.  Stephen  J.  Andriole,  Chairman 
Department  of  Information  Systems 
and  Systems  Engineering 
George  Mason  University 
4400  University  Drive 
Fairfax,  VA  22030 

Prof.  John  Annett 
University  of  Warwick 
Department  of  Psychology 
Coventry  CV4  7AL 
ENGLAND 

Edward  Atkins 
Code  61Z1210 

Naval  Sea  Systems  Command 
Washington,  DC  203(2-5101 

Dr.  Michael  E.  Atwood 
NYNEX 

At  I  alsAMS^AS 

500  Westchester  Avenue 
White  Plains,  NY  10604 

Dr.  Donald  E.  Bamber 
Code  440 

Naval  Ocean  Systems  Center 
San  Diego,  CA  92152-5000 


prof  doth  Bruno  G.  Bara 
Units  di  ricerca  di 
intdligenza  artificiale 
Universila  di  Milano 
20122  Milano  -  via  F.  Sfona  23 
rTALY 

Dr.  Jonathan  Baron 
80  Glenn  Avenue 
Berwyn,  PA  19312 

Dr.  Isaac  Bejar 
Law  School  Admissions 
Services 
P.O.  Bo*  40 

Newtown,  PA  189404XM0 

Leo  Bdtrscchi 
United  Stdtes  Nuclear 
Regulatory  Commission 
Washington  DC  205  S  5 

Dr.  Lawrence  Bimbaum 
The  Institute  for  the 
Learning  Sciences 
Northwestern  University 
1890  Maple  Avenue 
Evanston,  IL  60201 

Dr.  Gautam  Biswas 
Department  of  Computer  Science 
Box  1(88,  Station  B 
Vanderbilt  University 
Nashville,  TN  37235 

Dr.  John  Black 
Teachers  College,  Box  3 
Columbia  University 
525  West  120th  Street 
New  York,  NY  10027 

Dr.  Michael  Blackburn 
Code  943 

Naval  Ocean  Systems  Center 
San  Diego,  CA  92152-5000 

Dr.  Deborah  A.  Boehm-Dsvis 
Department  of  Psychology 
George  Mason  University 
4400  University  Drive 
Fairfax,  VA  22030 

Dr.  Sue  Bogner 

Army  Research  institute 

ATTN:  PER1-SF 

5001  Eiaenbcwer  Avenue 

Alexandria,  VA  22333-5(00 

Dr.  Jeff  Boost 
Guidance  Technology,  Inc 
800  Vrnial  Street 
Pittsburgh,  PA  15212 

Dr.  C  Alan  Bcneau 
Department  of  Psychology 
George  Mason  University 
4400  University  Drive 
FairfajgVA  22030 

Naval  Supply  Systems  Command 
NAVSUP  5512 
ATTN:  Sandra  Borden 
Washington,  D.C  20378-5000 

Dr.  J.  C  Boudreaux 
Center  for  Manufacturing 
Engineering 

N  sudds!  Bureau  of  Standards 
Gaithersburg  MD  20699 

Dr.  Lyle  E  Bourne,  Jr. 
Department  of  Psychology 
Box  345 

University  of  Colorado 
Boulder,  CO  80309 


Dr.  Robert  Breaux 
Code  281 

Naval  Training  Systems  Center 
Orlando,  FL  32828-3224 

Dr.  John  S.  Brown 
XEROX  Palo  Alto  Research 
Center 

3333  Coyote  Road 
Palo  Alto,  CA  94304 

Dr.  Hugh  Buma 
Department  of  English 
University  of  Texas 
Austin,  TX  78703 

Dr.  Robert  Calfee 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

Dr.  Joseph  C  Campione 
Center  for  the  Study  of  Reading 
University  of  Illinois 
51  Getty  Drive 
Champaign,  1L  (1820 

Dr.  Jaime  G.  Carbonell 
Computer  Science  Department 
Carnegie-Mellon  University 
Schenley  Park 
Pittsburgh,  PA  15213 

Dr.  Gail  Carpenter 
Center  for  Adaptive  Systems 
111  Cumminglon  St,  Room  244 
Boston  University 
Boston,  MA  02215 

Dr.  John  M.  Carroll 
IBM  Watson  Research  Center 
User  Interface  InaUtute 
P.O.  Box  704 

Yctktown  Heights.  NY  10598 

CDR  Robert  Carter 
Office  of  the  Chief 
of  Naval  Operations 
OP-933D4 

Washington,  DC  20350-2000 

Dr.  Tyrone  Casbman 
P.O.  Box  8129 
Minneapolis  MN  '55408 

Dr.  Fred  Chang 

Pacific  Bell 

2(00  Camino  Ramon 

Room  3S-450 

San  Ramon,  CA  94583 

Dr.  Davids  Cbansey 
English  Department 
Penn  State  University 
University  Park.  PA  1(802 

Dr.  Mkhelene  Chi 
Learning  R£D  Center 
University  of  Pittsburgh 
3939  OTiars  Street 
Pittsburgh,  PA  152(0 

Mrs.  Ola  Clarke 

818  South  George  Mason  Drive 

Arlington,  VA  22204 

Dr.  Norman  Cliff 
Department  or  psycnology 
Univ.  of  So.  California 
Los  Angeles,  CA  90089-10(1 

Dr.  Allan  M.  Collins 

Bolt  Beranek  &  Newman,  Inc. 

10  Moulton  Street 
Cambridge,  MA  02238 


Carnegie-Mdlon  Univenityflohn 


01/25/91 


Dr.  Sanity  Coilycr 
Offkx  of  Naval  Technology 
Cod e222 

800  N.  Quincy  Street 
Arlington,  VA  22217-5000 

Dr.  Meredith  P.  Crawford 
3583  Hamlet  Piece 
Chevy  Chare,  MD  20615 

Dr.  Hint  P.  Croat  beg 
Faculty  of  Law 
Univenity  of  Limburg 
P.O.  Bo*  616 
Maastricht 

The  NETHERLANDS  6200  MD 

Dr.  Kenneth  B.  Croat 
Anacapa  Science*,  Inc. 

P.O.  Drawer  Q 
Santa  Barbara,  CA  93102 

Dr.  Cary  Cticbon 
Intelligent  Instructional  Systems 
Teats  Instruments  AI  Lab 
P.O.  Box  660246 
Dallas,  TX  75266 

Mr.  John  F.  Dalphin 
Chair,  Computer  Science  Dept. 
Towaon  Sate  University 
Baltimore,  MD  21204 

Dr.  Gerald  F.  DeJong 
Computer  Science  Department 
University  of  Illinois 
Urbana,  IL  61801 

Goety  Ddacote 
Directeur  de  Llnfocmatique 
SdentiCque  et  Technique 
CNRS 

15,  Qual  Anatoie  France 
75700  Paris,  PRANCE 

Dr.  Denise  Dellarota 
Psychology  Department 
Box  HA,  Yale  Sution 
Yale  University 
New  Haven,  CT  06520-7447 

LT  COL  William  L.  Derrick 
HQ  USAF/MPXA 
The  Pentagon,  Rm.  5C360 
Washington,  DC  20330 

Dr.  Thomas  E.  DeZem 
Project  Engineer,  AI 
General  Dynamic* 

PO  Box  748/M  til  Zone  2646 
Fort  Worth,  TX  76101 

Dr.  Ronna  Dillon 
Department  i  Guidance  and 
Educational  Psychology 
Southern  IUioois  University 
Carbondale,  IL  62901 

Dr.  J.  Stuart  Donn 
Faculty  of  Education 
University  of  British  Columbia 
2125  Main  MaU 

Vancouver,  BC  CANADA  V6T  1Z5 

Defense  Technical 
Information  Center 
Cameron  Station.  Rida  5 
Alexandria,  VA  22314" 

(2  Copin) 

Dr.  Pierre  Duguct 
Organization  for  Economic 
Cooperation  and  Development 
2,  rue  Andre-Pascal 
75016  PARIS 
FRANCE 


Dr.  Ralph  Dusek 
V-P  Human  Factors 
JIL  Systems 

1225  Jefferson  Davit  Hwy. 

Suite  1209 

Arlington,  VA  22201 
Dr.  John  EUia 

Navy  Pettonne!  RAD  Center 
Code  51 

San  Diego,  CA  92252 

Dr.  Susan  Epstein 
144  S  Mountain  Avenue 
Montclair,  NJ  07042 

ERIC  Facility-Acquisitions 
2440  Research  Bhd,  Suite  550 
Rockville,  MD  20650-3238 

Dr.  K.  Anders  Ericsson 
University  of  Colorado 
Department  of  Piychology 
Campus  Box  345 
Boulder,  CO  803090345 

Dr.  Lorraine  D.  Eyde 
Office  of  Personnel  Management 
Office  of  Examination  Development 
1900  E  St,  NW 
Washington,  DC  20415 

LCDR  Michel ine  Y.  Eynud 
Code  602 

Naval  Air  Development  Center 
Warminster,  PA  18974-5000 

Dr.  Jean-Oaude  Falmagne 
Irvine  Research  Unit  in 
Mathematical  &  Behavioral  Science* 
University  of  California 
Irvine,  CA  92717 

Dr.  Marshall  J.  Farr,  Consulunt 
Cognitive  &  Instructional  Science* 
2520  North  Vernon  Street 
Arlington,  VA  22207 

Dr.  P-A.  Federico 
Code  51 
NPRDC 

San  Diego,  CA  921524800 

Dr.  Jerome  A.  Feldman 
Univenity  of  Rochester 
Computer  Science  Department 
Rochester,  NY  14627 

Dr.  Paul  Feltovicb 
Southern  Illinois  University 
School  of  Medicine 
Medical  Education  Department 
P.O.  Box  3926 
Springfield,  IL  62706 

CAPTJ.  Findli 
Commandant  (G-FTE) 

U.S.  Coast  Guard 
2100  Second  St,  SW. 

Washington,  DC  20593 

Prof.  Donald  Fitzgerald 
Univenity  of  New  England 
Department  of  Psychology 
Armidtle,  New  South  Wales  2351 
AUSTRALIA 

Dr.  Kenneth  D.  Focbus 
Institute  for  the  Learning 
Sciences 

Northwestern  Univenity 
1890  Ms  pie  Avenue 
Evanston,  IL  60201 


Dr.  Barbara  A.  Fox 
Univenity  of  Colorado 
Department  of  Languishes 
Boulder,  CO  80309 

Dr.  Csri  H.  Fredenkscn 
Dept,  of  Educational  Psychology 
McGill  Univenity 
3700  McTavitb  Street 
Montreal  Quebec 
CANADA  H3A  1Y2 

Dr.  John  R  Fredenkren 
BBN  Laboratories 
10  Moulton  Street 
Cambridge,  MA  02238 

Dr.  Norman  Fredenkscn 
Educational  Telling  Service 
(05-R) 

Princeton,  NJ  06541 

Dr.  Alfred  R  Fregly 
AFOSR/NL,  Bldg  410 
Bolling  AFB,  DC  203324443 

Dr.  Michael  Friendly 
Psychology  Department 
Yock  Univenity 
Toronto  ONT 
CANADA  M3J  1P3 

CoL  Dr.  Ernst  Frise 
Hecrespsycfaologiscbcf  Dienst 
Maria  Tbcresien-Kasemc 
1130  Wien 
AUSTRIA 

Dr.  Robert  M.  Gagne 
1456  Mitchell  Avenue 
Tallahassee,  FL  32303 

Dr.  Donald  R  Gen  her 
Philip*  Laboratories 
345  Scarborough  Road 
Briaidiff  Manor.  NY  10510 

Dr,  C  Lee  Giles 
AFOSR/NE,  Bldg  410 
Bolling  AFB 

Washington,  DC  20332 

Dr.  Philip  Giili* 

ARI-Fott  Gordon 
ATIN:  PERI-ICD 
Fort  Gordon,  GA  30905 

Mr.  Lee  Gladwin 
305  Davis  Avenue 
Leesburg  VA  22075 

Dr.  Robot  Glaser 
Learning  Research 
A  Development  Center 
Univenity  of  Pittsburgh 
3939  O’Hara  Street 
Pittsburgh,  PA  15260 

Dr.  Marvin  D.  Glock 
101  Homestead  Terrace 
Ithaca,  NY  14856 

Dr.  Dwight  J.  Gochring 
ARJ  Field  Unit 
P.O.  Box  5787 

Presidio  of  Monterey,  CA  93944-5011 

Dr.  Joseph  Gogucn 
Computer  Science  Laboratory 
SRI  International 
333  Rawenmood  Avenue 
Menlo  Part,  CA  94025 

Mr.  Richard  Golden 
Paycfaology  Department 
Stanford  Univenity 
Sanford  CA  94305 


Y 


J 


k 


L 


Camegie-Mellon  University/John 


Dr.  Timothy  Goldsmith 
Department  of  Psychology 
University  of  New  Mexico 
Albuquerque,  NM  87131 

Mr.  Harold  Goldstein 
University  of  DC 
Depertinent  Civil  Engineering 
Bldg.  42,  Room  IS 
4200  Connecticut  Avenue,  N.W. 
Washington,  DC  20006 

Dr.  Sherrie  Gou 
AFHRL/MOMJ 
Brooks  AFB,  TO  78235-5001 

Dr.  T.  Govindtrsj 
Georgia  Institute  of 
Technology 
School  of  Industrial 
and  Systems  Engineering 
Atlanta,  GA  303324205 

Dr.  Wayne  Gray 

Artificial  Intelligence  Laboratory 

NYNEX 

500  Westchester  Avenue 
White  Plains,  NY  10604 

Dr.  James  G.  Greeno 
School  of  Education 
Stanford  University 
Room  311 

Stanford,  CA  94305 

H.  William  Greenup 
Dep  Asst  C/S,  Instructional 
Management  (E03A) 

Education  Center,  MCCDC 
Quantko,  VA  22134-5050 

Dr.  Dik  Gregory 
Admiralty  Research 
EstaMiabmant/AXB 
Queens  Road 
Teddinston 

Middlesex,  ENGLAND  TW110LN 

Dr.  Stephen  Groasberg 
Center  for  Adaptive  Systems 
Room  244 

111  Cummington  Street 
Boston  University 
Boston,  MA  02215 

Michael  Ha  boo 
DORNIER  GMBH 
P.O.  Box  1420 
D-7990  Friedrichshafcn  1 
WEST  GERMANY 

Prof  Edward  Hacrtd 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

Dr.  Henry  M.  Halff 
HalfT  Resources,  Inc. 

4918  33rd  Road,  North 
Arlington,  VA  22207 

Mr,  H.  Hamburger 
Department  of  Computer  Science 
George  Mason  University 
Fairfax;  VA  22030 

Dr.  Bruce  W.  Hamid 
Research  Center 
The  Johns  Hopkins  University 
Applied  Physics  Laboratory 
Johns  Hopkins  Road 
Laurel,  MD  20707 

Dr.  Patrick  R.  Harrison 
Computer  Science  Department 
US  Naval  Academy 
Annapolis,  MO  21402-5002 


Dr.  Wayne  Harvey 

Center  for  Learning  Technology 

Education  Development  Center 

55  Chapel  Street 

Newton,  MA  02160 

Dr.  Barbara  Hayea-Roth 
Knowledge  Systems  Laboratory 
Stanford  University 
701  Welch  Road 
Palo  Alto,  CA  94304 

Dr.  Frederick  Hayes- Roth 
Teknowledge 
P.O.  Box  10119 
1850  Embarcadcro  Rd. 

Palo  Alto,  CA  94303 

Dr.  James  Hendler 
Dept,  of  Computer  Science 
University  of  Maryland 
College  Park,  MD  20742 

Dr.  James  Hiebert 
Department  of  Educational 
Development 
University  of  Delaware 
Newark,  DE  19716 

Dr.  Geoffrey  Hinton 

Computer  Science  Department 

University  of  Toronto 

Sandford  Fleming  Building 

W  King’s  College  Road 

Toronto,  Ontario  M5S  1A4  CANADA 

Dr.  Janus  E  Hoffman 
Department  of  Psychology 
University  of  Detain  re 
Newark,  DE  19711 

Dr,  Keith  Hotyoak 
Department  of  Psychology 
University  of  California 
Los  Angeles,  CA  90024 

Ms.  Julia  S.  Hough 
Cambridge  University  Press 
40  West  20th  Street 
New  Ycrit,  NY  10011 

Dr.  Wiliiam  Howell 
Chief  Scientist 
AJ’HRL/CA 

Brooks  AFB,  TO  78235-5601 

Dr.  Jack  Hunter 
2122  Cool  id  ge  Street 
Laming,  Ml  48906 

Dr.  Ed  Hutchins 
Intelligent  Systems  Group 
Institute  for 

Cognitive  Science  (0015) 

UCSD 

U  Jolla,  CA  92093 

Dr.  Claude  Janvier 
Univeraite*  du  Quebec  a  Montreal 
P.O.  Box  8888,  tucc:  A* 

Montreal,  Quebec  H3C  3P8 
CANADA 

Dr.  Robin  Jeffries 
Hewlett-Packard  Laboratories,  3L 
P.O.  Box  10490 
Palo  Alta  CA  943034971 

Dr.  Bonnie  E  John 
Wean  Had  8124 

Department  cf  Computer  Science 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 


Dr.  Peder  Johnson 
Department  of  Psychology 
University  of  New  Mexico 
Albuquerque,  NM  87131 

Dr.  Daniel  E  Jones 
US  Nuclear  Regulatory 
Commiasioa 
NRR/ILRB 

Washington.  DC  20555 

Mr.  Paul  L  Jones 

Research  Division 

Chief  of  Naval  Technical  Training 

Building  East-1 

Naval  Air  Station  Memphis 

Millington,  TO  38054-5056 

Mr.  Roland  Jones 
Mitre  Corp,  K-203 
Burlington  Road 
Bedford,  MA  01730 

Dr.  Ruth  Kinfer 
University  of  Minnesota 
Department  of  Psychology 
Elliott  Hall 
75  E  River  Road 
Minneapolis,  MN  55455 

Dr.  Michael  Kaplan 
Office  of  Bcaic  Research 
US.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333-5600 

Dr.  Demetrios  Karis 
GTE  Labs,  MS  61 
40  Sylvan  Road 
Waltham,  MA  02254 

Dr.  A.  Karmiloff-Smith 
MRC-CDU 
17  Gordon  Street 
Londoa 

ENGLAND  WC1H  OAH 

Dr.  Miltoa  S.  Kara 
European  Science  Coordination 
Office 

US  Army  Research  Institute 
Box  65 

FPO  New  Yotfc  09510-1500 

Dr.  Frank  Keil 
Department  of  Psychology 
228  Urit  Hall 
Cornell  University 
Ithaca,  NY  14850 

Dr.  Wendy  Kellogg 

IBM  T.  J.  Watson  Research  Ctr. 

P.O.  Box  794 

Yorktown  Heights,  NY  10596 

Dr.  Douglas  Kelly 
University  of  North  Carolina 
Department  of  Statistics 
Chapel  HilL  NC  27514 

Dr.  J.AS  Kelso 
Center  for  Complex  Systems 
Building  MT  9 
Florida  Atlantic  University 
Boca  Raton,  FL  33431 

Dr.  Rickard  Kerr 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  David  Kierae 

Technical  Communication  Program 
TIDAL  Bldg,  2360  Bonlstcd  BML 
University  of  Michigan 
Ann  Arbor,  MI  48109-2108 


Carnegie-Mellon  University/John 


Dr.  Thomas  Klllioa 
AFHRL/OT 

Williams  AFB,  AZ  &5 240-6457 

Dr.  Jeremy  Kilpatrick 
Department  of 

Mathematics  Education 
105  Adetbold  Hill 
University  o(  Georgia 
Athene.  GA  30602 

Dr.  Welter  Kintsch 
Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309434$ 

Dr.  Janet  L.  Kolodner 
Georgia  Inetituta  of  Technology 
School  of  Infocmetloo 
A  Computer  Scieoos 
Atlanta,  GA  30332 

Dr.  Stephen  Koeelyn 
Harvard  Univereity 
1236  William  Jamee  Hall 
33  Krtland  St. 

Cambridge,  MA  02138 

Dr.  Kenneth  Kotovsky 
Community  Coliege  of 
Allegheny  County 
906  Ridge  Avenue 
Pitubutgb,  PA  13212 

Dr.  Keith  Kramer 
HO  Lab,  Code  3 530 
Naval  Research  Laboratory 
♦MS  Overlook  Avenue 
Washington,  DC  20373-3000 

Dr.  Gary  Kress 
628  Spader  Avenue 
Pacific  Grove,  CA  93950 

Dr.  Benjamin  Kuipcrs 
University  of  Texes  at  Austin 
Department  of  Computer  Sciences 
Taylor  Hall  1124 
Austin,  Taras  78712 

Dr.  David  R.  Lambert 
Naval  Ocean  Systems  Center 
Code  772 

271  Catalina  Boulevard 
San  Diego,  CA  921313000 

Dr.  Pat  Langley 
NASA  Ames  Research  Ctr. 
Moffett  Field,  CA  9W3S 

Dr.  Jill  Larkin 
Camegie-Mclloo  University 
Department  of  Pqrchology 
Pittsburgh,  PA  15213 

Dr.  Robert  W.  Lawler 
Matthews  US 
Purdue  University 
West  Lafayette,  IN  47907 

Dr.  Eu  gene  Leo 
Naval  Postgraduate  School 
Monterey.  CA  93943-3026 

Dr.  Yub-Jeng  Lee 
Department  of  Computer  Science 
Code  32Le 

Naval  Postgraduate  School 

Monterey,  CA  93943 

Dr.  J  ill  P.  Lehman 
School  of  Computer  Science 
Carnegie  Mellon  University 
Pittsburgh,  PA  13213-3890 


Dr.  Alan  M.  La  gold 
Learning  RAD  Center 
University  of  Pittsburgh 
Pittsburgh,  PA  13260 

Dr.  Jim  Levin 
Department  of 
Educational  Psychology 
210  Education  Building 
1310  South  Sbtth  Street 
Champaign,  IL  618206990 

Dr.  John  Levine 
Learning  RAD  Center 
University  cf  Pittsburgh 
Pittsburgh,  PA  13260 

Dr.  Michael  Levine 
Educational  Psychology 
210  Education  Bldg 
University  of  Illinois 
Champaign.  IL  61801 

Matt  Lewis 

Department  of  Psychology 
Camegie-Mellon  University 
Pittsburgh,  PA  15213 

Library,  M.  Bremer 

Bolt,  Bcranek  sod  Newman,  Inc. 

K>  Moulton  Street 
Cambridge,  MA  02238 

Dr.  Doris  K.  Lidtke 
Software  Productivity  Consortium 
1880  Campus  Commons  Drive,  North 
Reston,  VA  22091 

Dr.  Marcia  C  Linn 
Graduate  School 
of  Education,  EMST 
Telman  Hall 
University  of  California 
Berkeley,  CA  94720 

Vem  M.  Maiec 

NPRDC,  Code  52 

San  Diego,  CA  921326800 

Dr.  Jaoa  Malm 
Mail  Code  EPS 
NASA  Johnson  Space  Center 
Houston,  TX  77058 

Dr.  William  L  Mtloy 
Code  04 
NETPMSA 

Pensacola.  PL  32309-5000 

Dr.  Mary  Mariino 

Director,  Educational  Technology 

HQUSAFA/DFTE 

USAF  Academy,  CO  80640-5000 

Dr.  Sandra  P.  Marshall 
Dept  of  Psychology 
San  Diego  State  University 
San  Diego,  CA  92182 

Dr.  John  H.  Muon 
Centre  for  Maths  Education 
Mathematics  Faculty 
Open  University 
Milton  Keynes  MK7  6AA 
UNITED  KINGDOM 

Dr.  Manton  M.  Matthews 
Department  of  Computer  Science 
University  of  South  Carolina 
Columbia,  SC  29206 

Dr.  Richard  B.  Mayer 
Department  of  Psychology 
University  of  California 
Santa  Barbara,  CA  93106 


Dr.  David  3.  McGuinneaa 
Gallaudct  University 
800  Florida  Avenue,  N.E. 
Washington,  DC  20002 

Dr.  Joseph  C  McLachlan 
Code  52 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92132-6800 

Dr.  Barbara  Meana 
SRI  International 
333  Ravenawood  Avenue 
Menlo  Park,  CA  94023 

Dr.  Douglu  L  Medin 
Department  of  Psychology 
University  of  Michlgsn 
Ann  Arbor,  Ml  48109 

Mr.  Stig  Meincke 
Forrvsrets  Center  for  Lcdcrskab 
Christianshavns  Voldgade  8 
142'  Kobenhavn  K 
DENMARK 

Dr.  Jose  Mestre 
Department  of  Physics 
Haabrouck  Laboratory 
University  of  Massachusetts 
Amherst,  MA  01003 

Dr.  Alan  Meyrowiti 
Office  of  Naval  Research 
Code  1123 
800  N.  Quincy 
Arlington,  VA  22217-5000 

Dr.  Ryaiard  X  MichiUti 
Center  for  Artificial  Intelligence 
George  Maaon  University 
Science  and  Tech  Room  301 
4400  University  Drive 
Fairfax  VA  22030-4444 

Dr.  D.  Michie 
The  Thring  Institute 
George  House 
36  North  Hanover  Street 
Glasgow  G1  2AD 
UNITED  KINGDOM 

Dr.  Vittorio  Midoro 
CNR-Istituto  Tecnotogie  Didatlicbc 
Via  All-Opera  Pia  11 
GENOVA-ITAL1A  16145 

Dr.  Jam a  R.  Miller 
MCC 

3500  W.  Baioones  Center  Dr. 
Austin,  TX  78759 

Dr.  Robert  Mialevy 
Educational  Toting  Service 
Princeton,  NJ  06541 

Dr.  Christine  M.  Mitchell 
School  of  Indus,  and  Sys.  Eng 
Center  for  Man-Mscbinc 
Systems  Research 
Georgia  Institute  of  Technology 
Atlanta,  GA  30532-0205 

Dr.  Andrew  R.  Mdnar 
AppHc.  of  Advanced  Technolop 
Science  and  Engr.  Education 
National  Srienne  Foundation 
Washington,  DC  20550 

Dr.  Melvin  D.  Montemerio 
NASA  Headquarters 
Code  RC 

Washington,  DC  20546 


01/25191 


1 


* 


i 


t 


Camegie-MeUon  Univenity/Jobn 


01/25/91 


Prof.  John  Morton 
MRC  Cognitive 
Development  Unit 
17  Gordon  Street 
London  WC1H  OAH 
UNrTED  KINGDOM 

Dr.  Randy  Mumaw 
Human  Science* 

Weatingbouae  Science 
&  Technology  Ctr. 

1310  Beulah  Roed 
Pittsburgh,  PA  15235 

Dr.  Allen  Munro 
Behavioral  Technology 
Lahocitodea  •  USC 
250  N.  Harbor  Dr,  Suite  309 
Redondo  Beach.  CA  90277 

Dr.  William  R.  Murray 
FMC  Corporation 
Central  Engineering  Labe 
1205  Coleman  Avenue 
Box  580 

Santa  data,  CA  95u52 

Chair,  Department  of  Weapon*  and 
Systems  Engineering 
U.S.  Naval  Academy 
Annapolis,  MD  21902 

Mr.  J.  Neliaten 

Taente  University  of  Technology 
Fee.  BibL  Toegepatte  Oodetwyskurdc 
P.  O.  Box  217 
7500  AE  Enacbede 
The  NETHERLANDS 

Dr.  T.  Niblett 
The  Hiring  Inatitute 
George  Houle 
3d  North  Hanover  Street 
Glasgow  G1  2AD 
UNITED  KINGDOM 

Dr.  A.  F.  Norrio 
Code  5530 

Navil  Reaearch  Laboratory 
Washington,  DC  20375-5000 

•>.  Donald  A.  Norman 
^015 

Inatitute  for  Cognitive  Science 
Univeraity  of  California 
La  Jolla.  CA  92093 

Libeaty,  NPRDC 
Code  P201L 

San  Dieyx  CA  92152-9800 
Librarian 

Naval  Center  for  Applied  Reaearch 
in  Artificial  Intelligence 
Naval  Reaearch  Laboratory 
Code  5510 

Washington,  DC  20375-5000 

Dr.  Harold  F.  O’Neil,  Jr. 

School  of  Education  -  WPH  801 
Department  of  Educational 
P*ychology  &  Technology 
Univeraity  of  Southern  California 
Loa  Angeiea,  CA  900099X131 

Dr.  Paul  CRorte 
Information  &  Computer  Science 
univeraity  or  California,  urine 
Irvine.  CA  92717 

Dr.  Stellar  Ohlaaon 
Learning  R  U  D  Center 
Univenity  of  Pittaburgh 
Pittaburgh,  PA  15290 


Dr.  Jamei  B.  Otaen 
WICAT  Syttem* 

1875  South  Slate  Street 
Orem,  UT  89058 

Dr.  Gaiy  M.  Oboo 
Cognitive  Science  and 
Machine  InteJUgence  Lab. 
Univeraity  of  Michigan 
701  Tappan  Street 
Ann  Arbor,  Ml  98109-1239 

Dr.  Judith  Reitman  Oboo 
Graduate  School  of  Business 
Univenity  of  Michigan 
Ann  Arbor,  Ml  98109-1239 

Office  of  Naval  Reaearch, 

Code  1192CS 
800  N.  Quincy  Street 
Arlington,  VA  22217-5000 
(6  Copiea) 

Dr.  Judith  Oruanu 
Baaic  Reaearch  Office 
Army  Reaearch  Inatitute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Jeaaa  Orta  ruby 
Inatitute  for  Defense  Analytes 
1801  N.  Beauregaid  St. 
Alexandria,  VA  22311 

Dr.  Everett  Palmer 
Mail  Stop  292-9 
NASA-Amea  Reaearch  Center 
Moffett  Field.  CA  99035 

Dr.  Okcboon  Part 
Army  Reaearch  Inatitute 
PERI-2 

5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Roy  Pea 
Inatitute  for  Reaearch 
on  Learning 
2550  Hanover  Street 
Palo  Alto,  CA  99309 

Dr.  Ray  S.  Perez 
ARI  (PERM!) 

5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  David  N.  Patina 
Project  Zero 

Hatvird  Graduate  School 
of  Educatioa 
7  Appiea  Way 
Cambridge,  MA  02138 

Dr.  C  Perrino,  Chair 
Dept,  of  Piychotogy 
Morgan  State  Univer 
Cold  Spring  La.-HiUer. 

Baltimore,  MD  21239 

Dept,  of  Adminiatrative  Sciences 
Code  59 

Navil  Postgraduate  School 
Monterey,  CA  93993-5029 

Dr.  Peter  Pirolli 
School  of  Education 
Univeraity  of  California 
Berkeley,  CA  99720 

Prof  Tomaso  Poggjo 
Massachusetts  inatitute 
of  Technology  E25-201 
Center  for  Biological 
Information  Processing 
Cambridge,  MA  02139 


Dr.  Peter  Pobon 
Univenity  of  Colorado 
Department  of  Psychology 
Boulder,  CO  803099X395 

Dr.  Steven  E  Poltrock 
Boeing  Advanced  Technology  Center 
PO  Box  24399  m/a  7L-64 
Seattle,  WA  98129 

Dr.  Joseph  Paotka 
ATTN:  PERMC 
Army  Research  Institute 
5001  Eisenhower  Ave. 

Alexandria,  VA  22333-5900 

Piyc  Info  -  CD  and  M 
American  Piycbdogical  Assoc. 

1200  Uble  Street 
Arlington,  VA  22201 

Mr.  Paul  S.  Rau 
Code  U-33 

Naval  Surface  Weapons  Center 
White  Oak  Laboratory 
Silver  Spring  MD  20903 

Dr.  Stephen  Redcr 
NWREL 

101  SW  Main.  Suite  500 
Portland,  OR  97209 

Dr.  James  A.  Reggja 
University  of  Marybnd 
School  of  Medicine 
Department  of  Neurology 
22  South  Greene  Street 
Baltimore,  MD  21201 

Dr.  J.  Wesley  Regjan 
AFHRL/IDI 

Brooks  AFB.TX  78235 

Dr.  Fred  Red 
Physics  Department 
Univetaity  of  California 
Berkeley.  CA  99720 

Dr.  Lauren  Resnick 
Learning  R  &  D  Center 
Univeraity  of  Pittaburgh 
3939  O’Hara  Surer 
Pittsburgh,  PA  15213 

Dr.  Gilbert  Ricard 
Mail  Stop  K02-19 
Grumman  Aircraft  Sy*tema 
Betbpage,  NY  11787 

Dr.  J.  Jeffrey  Richardson 
Center  for  Applied  A! 

College  of  Business 
Univetaity  of  Colorado 
Boulder,  CO  803099)919 

Dr.  Edwins  L  Riaaland 
Dept  of  Computer  and 
information  Science 
Univenity  of  Massachusetts 
Ambon,  MA  01003 

Dr.  Linda  G.  Roberta 
Scieocc,  Education,  and 
Transportation  Program 
Office  of  Technology  Aaaeaament 
Congreaa  of  the  United  States 
Washington,  DC  20510 

LT  CDR  Michael  N.  Rodger* 
Canadian  Forces  Personnel 
Applied  Research  Unit 
9900  Yonge  Street,  Suite  900 
Wiliowdale,  Ontario  M2N  9B7 
CANADA 


CaroegJe-MeUon  Unrvcrw'ty/John 


01/25/91 


Dr.  Ernst  Z.  Rolhkopf 

Dr.  Edward  Silver 

Dr.  Ted  Stetnkc 

AT&T  Beil  La  bon  tone* 

LRDC 

Dept,  of  Geography 

Room  2D-456 

Univenity  of  Pittahurgb 

Univenity  of  South  Carolina 

600  MounUio  Avenue 

3939  O’Hara  Street 

Columbia,  SC  292C6 

Murray  Hill,  NJ  07974 

Pittahurgb,  PA  15260 

Dr.  Ronald  Stemfeis 

Dr.  William  &  Room 

Dr.  Herbert  A  Simon 

Oak  Ridge  Assoc.  Univ. 

Search  Technology,  loo 

Department  of  Psychology 

P.O.  Box  117 

4725  Peachtree  Comen  Circle 

Carnegie-Melloo  Univenity 

Oak  Ridge,  TO  378314)117 

Suite  200 

ScbenleyPark 

Norcrose,  GA  30092 

Pittahurgb,  PA  15213 

Mr.  Michael  J.  Strait 

UMUC  Graduate  School 

Dr.  Fumiko  Samejima 

Robert  L  Simpson.  Jr. 

College  Park,  MD  20742 

Department  of  Paycbology 

DARPA/1STO 

University  of  Tennea.ee 

1400  Wihoo  Bbd. 

Dr.  Patrick  Suppea 

3106  Amlin  Peay  Bldg. 

Arlington,  VA  22209-2306 

Stanford  University 

Knoxville,  TO  37916-0900 

Institute  for  Mathematical 

Dr.  Zita  M.  Simutia 

Studies  in  the  Soda!  Sciences 

Dr.  Min  R  Scboenfekl 

Chief,  Technologies  for  Skill 

Stanford,  CA  94305-4115 

Univenity  o(  California 

Acquisition  and  Retention 

Department  of  Education 

ARl 

Dr.  Petty  W.  Tbomdyte 

Berkeley,  CA  94720 

5001  Eiaenbower  Avenue 

FMC  Corporation 

Alexandria,  VA  22333 

Central  Engineering  Labs 

Lowell  Scboer 

1205  Coleman  Avenue,  Box  580 

Paycboiogical  &  Quantitative 

Dr.  Derek  Sleeman 

Saau  Oars,  CA  95052 

Foundation. 

Computing  Science  Department 

College  of  Education 

The  Univenity 

Dr.  Douglas  Towne 

University  of  lowi 

Aberdeen  AB9  2FX 

Behavioral  Technology  Labe 

Iowa  Gty,  IA  52242 

Scotland 

Univenily  of  Southern  California 

UNITED  KINGDOM 

250  N.  Harbor  Dr,  Suite  309 

Dr.  Janet  W.  Schofield 

Redondo  Beach,  CA  90277 

816  LRDC  Building 

Ms.  Gad  K  Stetson 

Univenity  of  Pittahurgb 

LOGICON,  Inc. 

Major  D.  D.  Tucker 

3939  OTian  Street 

P.O.  Box  85158 

HQMC,  Code  MA  Room  4023 

Pittahurgb,  PA  15260 

San  Diego,  CA  92138-5158 

Washington.  DC  20380 

Dr.  Kay  Schulte 

Dr.  Edward  E.  Smith 

Dr.  Paul  T.  Twohig 

Computer  Science  Dept 

Department  of  Piycboto® 

Army  Research  Institute 

US  Naval  Academy 

University  of  Michigan 

ATTN:  PERI-RL 

Annapolis,  MD  214025018 

330  Packard  Road 

5001  Eisenhower  Avenue 

Dr.  Miriam  Schuatack 

Ann  Arbor,  Ml  48103 

Alexandria,  VA  22333-5600 

Code  52 

Dr.  Elliot  Soloway 

Dr.  Harold  P.  Van  Cou 

Navy  Pcnonnel  R  &  D  Center 

Yale  Univenity 

Committee  on  Human  Factor. 

San  Diego,  CA  921526800 

Computer  Science  Departmem 

National  Academy  of  Sciences 

P.O.  Box  2158 

2101  Constitution  Avenue 

Dr.  Colleen  M.  Seifert 

Department  of  Piycbology 

New  Haven.  CT  06S20 

Washington,  DC  20418 

Univenity  of  Michigan 

Linda  B.  Soriaio 

Dr.  Kurt  VanLehn 

330  Packard  Road 

lBM-Loa  Angeles  Scientific  Center 

Learning  Research 

Ann  Arbor.  MI  48104 

m  Wihhire  Btot,  4th  Floor 

ft  Development  Ctr. 

Df.  Michael  G.  Sbefto 

Loe  Angeles,  CA  90025 

Univenity  of  Pittsburgh 

3939  O’Hara  Street 

NASA  Amee  Reaearch  Ctr. 

Dr.  Kathryn  T.  Spoefar 

Pittsburgh,  PA  15260 

Mail  Stop  239-1 

Department  of  Cognitive 

Moffett  Field,  CA  94035 

&  Linguistic  Science 

Dr.  Diana  Weame 

Box  1978 

Department  of  Educational 

Dr.  Va>erie  L.  Sbalin 

Bream  Univenity 

Development 

Department  of  Industrial 

Providence,  R1  02912 

University  of  Delaware 

Engineering 

Newart.  DE  19711 

State  Univenity  of  New  York 

N.  S.  Sodbaran 

342  Lawrence  D.  Bell  Hall 

FMC  Corporation 

Dr.  Kolb  T.  Wescourt 

Buffalo,  NY  14260 

Box  580 

FMC  Corporation 

1205  Coleman  Avenue 

Central  Engineering  Labs 

Mr.  Colin  Sheppard 

Santa  Clara,  CA  95052 

1205  Coleman  Avt,  Box  580 

AXC2  Block  3 

Santa  Clara,  CA  95052 

Admirality  Reaearch  Establishment 

Dr.  James  J.  Stwtcwaki 

Minhtry  of  Defence  Pottadown 

Dept,  of  Psychology 

Dr.  David  Wilkint 

Portsmouth  Hanu  P064AA 

University  of  South  Carolina 

University  of  Illinois 

UNITED  KINGDOM 

Columbia,  SC  29210 

Department  of  Computer  Science 
1304  West  Springfield  Avenue 

Dr.  Ben  Shneidennan 

Dr.  Friedrich  W.  Steep: 

Uibaru,  IL  61801 

Dept,  of  Computer  Science 

Bundeaminiaterium 

Univenity  of  Maryland 

dea  VcrtcidJguag 

Dr.  Manba  R.  Williams 

College  Park,  MD  20742 

Pottfacfa  1328 

Apptic.  of  Advanced  Technologies 

D-5300  Bonn  1 

National  Science  Foundation 

Dr.  Lee  &  Sbulmao 

WEST  GERMANY 

SEE/MDRISE 

ococoi  ot  cr.ucstxxi 

1800  G  Struct,  N.W,  Room  635-A 

507  Cerae 

Dr.  Bruce  D.  Steinberg 

Washington,  DC  20550 

Stanford  University 

Curry  College 

Stanford,  CA  94305-3084 

Milton,  MA  02186 

Dr.  Mark  Wilson 

School  of  Education 

Dr.  Randall  Shumaker 

Dr.  Frederick  Steinbeiaer 

Univeraity  of  California 

Naval  Reaearch  Laboratory 

C1A-ORD 

Berkeley,  CA  94720 

Code  5510 

Amea  Building 

4555  Overlook  Avenue,  S.W. 

Wellington,  DC  20075-5000 

Washington,  DC  20505 

Camepe-Mdkxi  Univcnity/Jotin 


01/25/91 


SIR  Wileon 
Code  5505 

Nml  Reeearch  Laboratory 
Waehingtoo.  DC  20375-5000 

Mr.  Joeeph  Wohl 
Aiphatech,  Inc 

2  Burlington  Executive  Center 
111  Middlesex  Turnpike 
Burlington.  MA  01803 

Dr.  Mteoud  Yazdeni 
Dept  of  Computer  Science 
►  Univenity  of  Exeter 

Prince  of  Wales  Road 
Exeter  EX44PT 
ENGLAND 

b-  Dr.  Joeeph  L  Young 

Netiooal  Science  Foundation 

Room  320 

1800  G  Street  N.W. 

Waehingtoo,  DC  20550 

Dr.  Uri  Zemik 

General  Electric 

Reeearii  St  Development  Center 

Artificial  Intelligence  Program 

POBoxS 

Schenectady,  NY  12301 


1 


I 


